---
title: "Findings of the 2015 Workshop on Statistical — Bojar 2015"
description: "This report presents the results of the WMT15 evaluation campaign, held with EMNLP 2015 in Lisbon."
authors: ["Ondřej Bojar", "Rajen Chatterjee", "Christian Federmann", "Barry Haddow", "Matthias Huck", "Chris Hokamp", "Philipp Koehn", "Varvara Logacheva", "Christof Monz", "Matteo Negri", "Matt Post", "Carolina Scarton", "Lucia Specia", "Marco Turchi"]
year: 2015
venue: "Proceedings of the Tenth Workshop on Statistical Machine Translation"
source: "https://aclanthology.org/W15-3001/"
doi: "10.18653/v1/w15-3001"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Findings of the 2015 Workshop on Statistical Machine Translation

# Findings of the 2015 Workshop on Statistical Machine Translation

**Authors:** [Ondřej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/), [Rajen Chatterjee](https://md.research.elamur.ai/free/en/authors/rajen-chatterjee/), [Christian Federmann](https://md.research.elamur.ai/free/en/authors/christian-federmann/), [Barry Haddow](https://md.research.elamur.ai/free/en/authors/barry-haddow/), [Matthias Huck](https://md.research.elamur.ai/free/en/authors/matthias-huck/), [Chris Hokamp](https://md.research.elamur.ai/free/en/authors/chris-hokamp/), [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/), [Varvara Logacheva](https://md.research.elamur.ai/free/en/authors/varvara-logacheva/), [Christof Monz](https://md.research.elamur.ai/free/en/authors/christof-monz/), [Matteo Negri](https://md.research.elamur.ai/free/en/authors/matteo-negri/), [Matt Post](https://md.research.elamur.ai/free/en/authors/matt-post/), [Carolina Scarton](https://md.research.elamur.ai/free/en/authors/carolina-scarton/) et al. (14 authors) · **Year:** 2015 · **Venue:** Proceedings of the Tenth Workshop on Statistical Machine Translation · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W15-3001/) · [PDF](https://aclanthology.org/W15-3001.pdf) · [DOI 10.18653/v1/w15-3001](https://doi.org/10.18653/v1/w15-3001) · [OpenAlex](https://openalex.org/W2251994258)

**Citations:** 277 citations (openalex-snapshot · 2026-10-14) · [184](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This report presents the results of the WMT15 evaluation campaign, held with EMNLP 2015 in Lisbon. It measures the current state of machine translation and of tools that judge or repair MT output. The campaign had five tasks: news translation, metrics, tuning, quality estimation (QE) and a first, pilot automatic post-editing (APE) task. The metrics and tuning tasks are described in separate papers. The translation task covered English paired with Czech, French, German, Russian and Finnish, in both directions. Finnish took Hindi's place as the harder, less-resourced language. Organizers supplied training corpora and newly translated test sets. Systems were ranked through large-scale human judgments, aggregated with TrueSkill, and through automatic metrics. The APE pilot gave a clear lesson: no participant beat a baseline that simply leaves the MT output unchanged. Current statistical APE systems tend to over-correct. The campaign matters because all test data, system outputs and human judgments are released publicly, giving shared benchmarks for later MT and evaluation research.

## Key points
- The translation task received 68 systems from 24 institutions across ten directions. Seven anonymized external systems were added (1 commercial, 6 online) and counted as unconstrained.
- Test sets had about 3000 sentences per pair: roughly 1500 originally in English and 1500 originally in the other language. The French–English set instead used reader comments from the Guardian and Le Monde (38 threads each).
- Human ranking was streamlined by removing near-identical candidate outputs before annotation. TrueSkill produced the final system ordering.
- QE drew 10 teams and 34 entries across three subtasks. These were sentence-level post-editing effort (English→Spanish), word-level good/bad labels (English→Spanish) and document-level Meteor prediction (English↔German).
- The APE pilot attracted 4 teams and 7 entries on English→Spanish news, using data from Unbabel. Systems beat plain monolingual translation but not the "do nothing" baseline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

184 papers of our library cite it (of the 140884 whose bibliographies we read); 30 of them have a page here, 154 not yet.

- [Findings of the 2021 Conference on Machine Translation (WMT21)](https://md.research.elamur.ai/free/en/papers/akhbardeh-2021-findings-2021-conference-machine-translation/)
- [Findings of the 2019 Conference on Machine Translation (WMT19)](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
- [Findings of the 2020 Conference on Machine Translation (WMT20)](https://md.research.elamur.ai/free/en/papers/barrault-2020-findings-2020-conference-machine-translation/)
- [Neural versus Phrase-Based Machine Translation Quality: a Case Study](https://md.research.elamur.ai/free/en/papers/bentivogli-2016-neural-versus-phrase-based-machine/)
- [HUME: Human UCCA-Based Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/birch-2016-hume-human-ucca-based-evaluation/)
- [Findings of the 2016 Conference on Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/)
- [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/)
- [Findings of the 2018 Conference on Machine Translation (WMT18)](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/)
- [An Open Dataset and Model for Language Identification](https://md.research.elamur.ai/free/en/papers/burchell-2023-open-dataset-model-language-identification/)
- [Findings of the WMT 2018 Shared Task on Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/chatterjee-2018-findings-wmt-2018-shared-task/)
- [Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/dowmunt-2016-log-linear-combinations-monolingual-bilingual/)
- [Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation](https://md.research.elamur.ai/free/en/papers/ferreira-2017-linguistic-realisation-as-machine-translation/)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://md.research.elamur.ai/free/en/papers/hokamp-2017-lexically-constrained-decoding-sequence-generation/)
- [GlotLID: Language Identification for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/kargaran-2023-glotlid-language-identification-low-resource/)
- [Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/)
- [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/en/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
- [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://md.research.elamur.ai/free/en/papers/libovicky-2018-end-end-non-autoregressive-neural/)
- [MEANT 2.0: Accurate semantic MT evaluation for any output language](https://md.research.elamur.ai/free/en/papers/lo-2017-meant-2-0-accurate-semantic/)
- [Predicting and Using a Pragmatic Component of Lexical Aspect of Simple Past Verbal Tenses for Improving English-to-French Machine Translation](https://md.research.elamur.ai/free/en/papers/loaiciga-2016-predicting-pragmatic-component-lexical-aspect/)
- [Multi-task Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
- [DTED: Evaluation of Machine Translation Structure Using Dependency Parsing and Tree Edit Distance](https://md.research.elamur.ai/free/en/papers/mccaffery-2016-dted-evaluation-machine-translation-structure/)
- [SemiAdapt: Semi-Supervised and Efficient LoRA-Based Domain Adaptation for Low-Resource Irish Machine Translation with Transformers](https://md.research.elamur.ai/free/en/papers/mcgiff-2026-semiadapt-semi-supervised-efficient-lora/)
- [Investigating Backtranslation in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/)
- [Style Transfer Through Back-Translation](https://md.research.elamur.ai/free/en/papers/prabhumoye-2018-style-transfer-through-back-translation/)
- [Unsupervised Pretraining for Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/ramachandran-2017-unsupervised-pretraining-sequence-sequence-learning/)
- [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://md.research.elamur.ai/free/en/papers/rubenstein-2023-audiopalm-large-language-model-that/)
- [A Shared Task on Multimodal Machine Translation and Crosslingual Image Description](https://md.research.elamur.ai/free/en/papers/specia-2016-shared-task-multimodal-machine-translation/)
- [Findings of the WMT 2018 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/en/papers/specia-2018-findings-wmt-2018-shared-task/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bojar2015findings,
  title = {{Findings of the 2015 Workshop on Statistical Machine Translation}},
  author = {Bojar, Ondřej and Chatterjee, Rajen and Federmann, Christian and Haddow, Barry and Huck, Matthias and Hokamp, Chris and Koehn, Philipp and Logacheva, Varvara and Monz, Christof and Negri, Matteo and Post, Matt and Scarton, Carolina and Specia, Lucia and Turchi, Marco},
  booktitle = {Proceedings of the Tenth Workshop on Statistical Machine Translation},
  year = {2015},
  doi = {10.18653/v1/w15-3001},
  url = {https://doi.org/10.18653/v1/w15-3001}
}
```

- [← previous Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/)
- [next → Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings](https://md.research.elamur.ai/free/en/papers/liu-2015-fine-grained-opinion-mining-recurrent/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

86 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Avramidis, E., Popović, M., and Burchardt, A. (2015). DFKI’s experimental hybrid MT system for WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 66–73, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Avramidis%2C+E%2E%2C+Popovic%CC%81%2C+M%2E%2C+and+Burchardt%2C+A%2E+%282015%29%2E+DFKI%E2%80%99s+experimental+hybrid+MT+system+for+WMT+2015%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+66%E2%80%9373%2C+Lisboa%2C+)
2. Banerjee, S. and Lavie, A. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization. [find](https://scholar.google.com/scholar?q=Banerjee%2C+S%2E+and+Lavie%2C+A%2E+%282005%29%2E+METEOR%3A+An+Automatic+Metric+for+MT+Evaluation+with+Improved+Correlation+with+Human+Judgments%2E+In+Proceedings+of+the+ACL+2005+Workshop+on+Intrinsic+and+Extrinsic+Eval) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
3. Béchara, H., Ma, Y., and van Genabith, J. (2011). Statistical Post-Editing for a Statistical MT System. In Proceedings of the 13th Machine Translation Summit, pages 308–315, Xiamen, China. [find](https://scholar.google.com/scholar?q=Be%CC%81chara%2C+H%2E%2C+Ma%2C+Y%2E%2C+and+van+Genabith%2C+J%2E+%282011%29%2E+Statistical+Post%2DEditing+for+a+Statistical+MT+System%2E+In+Proceedings+of+the+13th+Machine+Translation+Summit%2C+pages+308%E2%80%93315%2C+Xiamen%2C+China%2E)
4. Beck, D., Shah, K., Cohn, T., and Specia, L. (2013a). SHEF-Lite: When less is more for translation quality estimation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 335–340, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Beck%2C+D%2E%2C+Shah%2C+K%2E%2C+Cohn%2C+T%2E%2C+and+Specia%2C+L%2E+%282013a%29%2E+SHEF%2DLite%3A+When+less+is+more+for+translation+quality+estimation%2E+In+Proceedings+of+the+Eighth+Workshop+on+Statistical+Machine+Translation%2C+pages+3)
5. Beck, D., Specia, L., and Cohn, T. (2013b). Reducing annotation effort for quality estimation via active learning. In 51st Annual Meeting of the Association for Computational Linguistics: Short Papers, ACL, pages 543–548, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Beck%2C+D%2E%2C+Specia%2C+L%2E%2C+and+Cohn%2C+T%2E+%282013b%29%2E+Reducing+annotation+effort+for+quality+estimation+via+active+learning%2E+In+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Short+Papers)
6. Biçici, E. (2013). Referential translation machines for quality estimation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Bic%CC%A7ici%2C+E%2E+%282013%29%2E+Referential+translation+machines+for+quality+estimation%2E+In+Proceedings+of+the+Eighth+Workshop+on+Statistical+Machine+Translation%2C+Sofia%2C+Bulgaria%2E)
7. Biçici, E. and Way, A. (2014). Referential Translation Machines for Predicting Translation Quality. In Ninth Workshop on Statistical Machine Translation, pages 313–321, Baltimore, Maryland, USA. [find](https://scholar.google.com/scholar?q=Bic%CC%A7ici%2C+E%2E+and+Way%2C+A%2E+%282014%29%2E+Referential+Translation+Machines+for+Predicting+Translation+Quality%2E+In+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+313%E2%80%93321%2C+Baltimore%2C+Maryland%2C+USA%2E)
8. Bicici, E., Liu, Q., and Way, A. (2015). Referential Translation Machines for Predicting Translation Quality and Related Statistics. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 304–308, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bicici%2C+E%2E%2C+Liu%2C+Q%2E%2C+and+Way%2C+A%2E+%282015%29%2E+Referential+Translation+Machines+for+Predicting+Translation+Quality+and+Related+Statistics%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Transla)
9. Blain, F., Schwenk, H., and Senellart, J. (2012). Incremental adaptation using translation information and post-editing analysis. In International Workshop on Spoken Language Translation (IWSLT), pages 234–241, Hong-Kong (China). [find](https://scholar.google.com/scholar?q=Blain%2C+F%2E%2C+Schwenk%2C+H%2E%2C+and+Senellart%2C+J%2E+%282012%29%2E+Incremental+adaptation+using+translation+information+and+post%2Dediting+analysis%2E+In+International+Workshop+on+Spoken+Language+Translation+%28IWSLT%29%2C+page)
10. Bojar, O., Buck, C., Callison-Burch, C., Federmann, C., Haddow, B., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L. (2013). [find](https://scholar.google.com/scholar?q=Bojar%2C+O%2E%2C+Buck%2C+C%2E%2C+Callison%2DBurch%2C+C%2E%2C+Federmann%2C+C%2E%2C+Haddow%2C+B%2E%2C+Koehn%2C+P%2E%2C+Monz%2C+C%2E%2C+Post%2C+M%2E%2C+Soricut%2C+R%2E%2C+and+Specia%2C+L%2E+%282013%29%2E)
11. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1–42, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Findings+of+the+2013+Workshop+on+Statistical+Machine+Translation%2E+In+Proceedings+of+the+Eighth+Workshop+on+Statistical+Machine+Translation%2C+pages+1%E2%80%9342%2C+Sofia%2C+Bulgaria%2E+Association+for+Computational+L) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2013-findings-2013-workshop-statistical-machine/)
12. Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. (2014). Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12– 58, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bojar%2C+O%2E%2C+Buck%2C+C%2E%2C+Federmann%2C+C%2E%2C+Haddow%2C+B%2E%2C+Koehn%2C+P%2E%2C+Leveling%2C+J%2E%2C+Monz%2C+C%2E%2C+Pecina%2C+P%2E%2C+Post%2C+M%2E%2C+Saint%2DAmand%2C+H%2E%2C+Soricut%2C+R%2E%2C+Specia%2C+L%2E%2C+and+Tamchyna%2C+A%2E+%282014%29%2E+Findings+of+the+2014+worksho) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
13. Bojar, O. and Tamchyna, A. (2015). CUNI in WMT15: Chimera Strikes Again. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 79–83, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bojar%2C+O%2E+and+Tamchyna%2C+A%2E+%282015%29%2E+CUNI+in+WMT15%3A+Chimera+Strikes+Again%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+79%E2%80%9383%2C+Lisboa%2C+Portugal%2E+Association+for+Computa)
14. Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., and Schroeder, J. (2007). (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT07), Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Callison%2DBurch%2C+C%2E%2C+Fordyce%2C+C%2E%2C+Koehn%2C+P%2E%2C+Monz%2C+C%2E%2C+and+Schroeder%2C+J%2E+%282007%29%2E+%28Meta%2D%29+evaluation+of+machine+translation%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+Translation+%28WMT) [our card](https://md.research.elamur.ai/free/en/papers/burch-2007-meta-evaluation-machine-translation/)
15. Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., and Schroeder, J. (2008). Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08), Colmbus, Ohio. [find](https://scholar.google.com/scholar?q=Callison%2DBurch%2C+C%2E%2C+Fordyce%2C+C%2E%2C+Koehn%2C+P%2E%2C+Monz%2C+C%2E%2C+and+Schroeder%2C+J%2E+%282008%29%2E+Further+meta%2Devaluation+of+machine+translation%2E+In+Proceedings+of+the+Third+Workshop+on+Statistical+Machine+Translation+) [our card](https://md.research.elamur.ai/free/en/papers/burch-2007-meta-evaluation-machine-translation/)
16. Callison-Burch, C., Koehn, P., Monz, C., Peterson, K., Przybocki, M., and Zaidan, O. F. (2010). Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT10), Uppsala, Sweden. [find](https://scholar.google.com/scholar?q=Callison%2DBurch%2C+C%2E%2C+Koehn%2C+P%2E%2C+Monz%2C+C%2E%2C+Peterson%2C+K%2E%2C+Przybocki%2C+M%2E%2C+and+Zaidan%2C+O%2E+F%2E+%282010%29%2E+Findings+of+the+2010+joint+workshop+on+statistical+machine+translation+and+metrics+for+machine+translati) [our card](https://md.research.elamur.ai/free/en/papers/burch-2010-findings-2010-joint-workshop-statistical/)
17. Callison-Burch, C., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L. (2012). Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10– 51, Montréal, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Callison%2DBurch%2C+C%2E%2C+Koehn%2C+P%2E%2C+Monz%2C+C%2E%2C+Post%2C+M%2E%2C+Soricut%2C+R%2E%2C+and+Specia%2C+L%2E+%282012%29%2E+Findings+of+the+2012+workshop+on+statistical+machine+translation%2E+In+Proceedings+of+the+Seventh+Workshop+on+Stati) [our card](https://md.research.elamur.ai/free/en/papers/burch-2012-findings-2012-workshop-statistical-machine/)
18. Callison-Burch, C., Koehn, P., Monz, C., and Schroeder, J. (2009). Findings of the 2009 workshop on statistical machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT09), Athens, Greece. [find](https://scholar.google.com/scholar?q=Callison%2DBurch%2C+C%2E%2C+Koehn%2C+P%2E%2C+Monz%2C+C%2E%2C+and+Schroeder%2C+J%2E+%282009%29%2E+Findings+of+the+2009+workshop+on+statistical+machine+translation%2E+In+Proceedings+of+the+Fourth+Workshop+on+Statistical+Machine+Transl) [our card](https://md.research.elamur.ai/free/en/papers/burch-2009-findings-2009-workshop-statistical-machine/)
19. Callison-Burch, C., Koehn, P., Monz, C., and Zaidan, O. (2011). Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland. [find](https://scholar.google.com/scholar?q=Callison%2DBurch%2C+C%2E%2C+Koehn%2C+P%2E%2C+Monz%2C+C%2E%2C+and+Zaidan%2C+O%2E+%282011%29%2E+Findings+of+the+2011+workshop+on+statistical+machine+translation%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translatio) [our card](https://md.research.elamur.ai/free/en/papers/burch-2011-findings-2011-workshop-statistical-machine/)
20. Camargo de Souza, J. G., González-Rubio, J., Buck, C., Turchi, M., and Negri, M. (2014). Fbk-upv-uedin participation in the wmt14 quality estimation shared-task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 322–328, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Camargo+de+Souza%2C+J%2E+G%2E%2C+Gonza%CC%81lez%2DRubio%2C+J%2E%2C+Buck%2C+C%2E%2C+Turchi%2C+M%2E%2C+and+Negri%2C+M%2E+%282014%29%2E+Fbk%2Dupv%2Duedin+participation+in+the+wmt14+quality+estimation+shared%2Dtask%2E+In+Proceedings+of+the+Ninth+Workshop+)
21. Cap, F., Weller, M., Ramm, A., and Fraser, A. (2015). CimS - The CIS and IMS Joint Submission to WMT 2015 addressing morphological and syntactic differences in English to German SMT. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 84–91, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cap%2C+F%2E%2C+Weller%2C+M%2E%2C+Ramm%2C+A%2E%2C+and+Fraser%2C+A%2E+%282015%29%2E+CimS+%2D+The+CIS+and+IMS+Joint+Submission+to+WMT+2015+addressing+morphological+and+syntactic+differences+in+English+to+German+SMT%2E+In+Proceedings+of)
22. Cettolo, M., Bertoldi, N., and Federico, M. (2014). The Repetition Rate of Text as a Predictor of the Effectiveness of Machine Translation Adaptation. In Proceedings of the 11th Biennial Conference of the Association for Machine Translation in the Americas (AMTA 2014), pages 166– 179, Vancouver, BC, Canada. [find](https://scholar.google.com/scholar?q=Cettolo%2C+M%2E%2C+Bertoldi%2C+N%2E%2C+and+Federico%2C+M%2E+%282014%29%2E+The+Repetition+Rate+of+Text+as+a+Predictor+of+the+Effectiveness+of+Machine+Translation+Adaptation%2E+In+Proceedings+of+the+11th+Biennial+Conference+of)
23. Chatterjee, R., Turchi, M., and Negri, M. (2015a). The FBK Participation in the WMT15 Automatic Post-editing Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 210–215, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chatterjee%2C+R%2E%2C+Turchi%2C+M%2E%2C+and+Negri%2C+M%2E+%282015a%29%2E+The+FBK+Participation+in+the+WMT15+Automatic+Post%2Dediting+Shared+Task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages)
24. Chatterjee, R., Weller, M., Negri, M., and Turchi, M. (2015b). Exploring the Planet of the APEs: a Comparative Study of State-of-the-art Methods for MT Automatic Post-Editing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics), Beijing, China. [find](https://scholar.google.com/scholar?q=Chatterjee%2C+R%2E%2C+Weller%2C+M%2E%2C+Negri%2C+M%2E%2C+and+Turchi%2C+M%2E+%282015b%29%2E+Exploring+the+Planet+of+the+APEs%3A+a+Comparative+Study+of+State%2Dof%2Dthe%2Dart+Methods+for+MT+Automatic+Post%2DEditing%2E+In+Proceedings+of+the+53)
25. Cho, E., Ha, T.-L., Niehues, J., Herrmann, T., Mediani, M., Zhang, Y., and Waibel, A. (2015). The Karlsruhe Institute of Technology Translation Systems for the WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 92–97, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cho%2C+E%2E%2C+Ha%2C+T%2E%2DL%2E%2C+Niehues%2C+J%2E%2C+Herrmann%2C+T%2E%2C+Mediani%2C+M%2E%2C+Zhang%2C+Y%2E%2C+and+Waibel%2C+A%2E+%282015%29%2E+The+Karlsruhe+Institute+of+Technology+Translation+Systems+for+the+WMT+2015%2E+In+Proceedings+of+the+Tenth+Wo)
26. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurment, 20(1):37–46. [↗](https://doi.org/10.1177/001316446002000104)
27. Dušek, O., Gomes, L., Novák, M., Popel, M., and Rosa, R. (2015). New Language Pairs in TectoMT. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 98– 104, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dus%CC%8Cek%2C+O%2E%2C+Gomes%2C+L%2E%2C+Nova%CC%81k%2C+M%2E%2C+Popel%2C+M%2E%2C+and+Rosa%2C+R%2E+%282015%29%2E+New+Language+Pairs+in+TectoMT%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+98%E2%80%93+104%2C+Lisboa%2C+Portug)
28. Dyer, C., Lopez, A., Ganitkevitch, J., Weese, J., Ture, F., Blunsom, P., Setiawan, H., Eidelman, V., and Resnik, P. (2010). cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In Proceedings of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Dyer%2C+C%2E%2C+Lopez%2C+A%2E%2C+Ganitkevitch%2C+J%2E%2C+Weese%2C+J%2E%2C+Ture%2C+F%2E%2C+Blunsom%2C+P%2E%2C+Setiawan%2C+H%2E%2C+Eidelman%2C+V%2E%2C+and+Resnik%2C+P%2E+%282010%29%2E+cdec%3A+A+decoder%2C+alignment%2C+and+learning+framework+for+finite%2Dstate+and+cont) [our card](https://md.research.elamur.ai/free/en/papers/dyer-2010-cdec-decoder-alignment-learning-framework/)
29. Esplà-Gomis, M., Sánchez-Martı́nez, F., and Forcada, M. (2015a). UAlacant word-level machine translation quality estimation system at WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 309–315, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Espla%CC%80%2DGomis%2C+M%2E%2C+Sa%CC%81nchez%2DMart%C4%B1%CC%81nez%2C+F%2E%2C+and+Forcada%2C+M%2E+%282015a%29%2E+UAlacant+word%2Dlevel+machine+translation+quality+estimation+system+at+WMT+2015%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Ma)
30. Esplà-Gomis, M., Sánchez-Martnez, F., and Forcada, M. L. (2015b). Using on-line available sources of bilingual information for word-level machine translation quality estimation. In 18th Annual Conference of the European Associtation for Machine Translation, page 1926, Antalya, Turkey. [find](https://scholar.google.com/scholar?q=Espla%CC%80%2DGomis%2C+M%2E%2C+Sa%CC%81nchez%2DMartnez%2C+F%2E%2C+and+Forcada%2C+M%2E+L%2E+%282015b%29%2E+Using+on%2Dline+available+sources+of+bilingual+information+for+word%2Dlevel+machine+translation+quality+estimation%2E+In+18th+Annual+Confe)
31. Federmann, C. (2012). Appraise: An Open-Source Toolkit for Manual Evaluation of Machine Translation Output. The Prague Bulletin of Mathematical Linguistics (PBML), 98:25– 35. [find](https://scholar.google.com/scholar?q=Federmann%2C+C%2E+%282012%29%2E+Appraise%3A+An+Open%2DSource+Toolkit+for+Manual+Evaluation+of+Machine+Translation+Output%2E+The+Prague+Bulletin+of+Mathematical+Linguistics+%28PBML%29%2C+98%3A25%E2%80%93+35%2E)
32. Gao, Q. and Vogel, S. (2008). Parallel Implementations of Word Alignment Tool. In Proceedings of the ACL 2008 Software Engineering, Testing, and Quality Assurance Workshop, pages 49–57, Columbus, Ohio. [find](https://scholar.google.com/scholar?q=Gao%2C+Q%2E+and+Vogel%2C+S%2E+%282008%29%2E+Parallel+Implementations+of+Word+Alignment+Tool%2E+In+Proceedings+of+the+ACL+2008+Software+Engineering%2C+Testing%2C+and+Quality+Assurance+Workshop%2C+pages+49%E2%80%9357%2C+Columbus%2C+Ohio) [our card](https://md.research.elamur.ai/free/en/papers/gao-2008-parallel-implementations-word-alignment-tool/)
33. Giménez, J. and Màrquez, L. (2010). Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation. The Prague Bulletin of Mathematical Linguistics, 94:77–86. [find](https://scholar.google.com/scholar?q=Gime%CC%81nez%2C+J%2E+and+Ma%CC%80rquez%2C+L%2E+%282010%29%2E+Asiya%3A+An+Open+Toolkit+for+Automatic+Machine+Translation+%28Meta%2D%29Evaluation%2E+The+Prague+Bulletin+of+Mathematical+Linguistics%2C+94%3A77%E2%80%9386%2E)
34. Graham, Y. (2015). Improving Evaluation of Machine Translation Quality Estimation. In 53rd Annual Meeting of the Association for Computational Linguistics and Seventh International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, pages 1804–1813, Beijing, China. [find](https://scholar.google.com/scholar?q=Graham%2C+Y%2E+%282015%29%2E+Improving+Evaluation+of+Machine+Translation+Quality+Estimation%2E+In+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+Seventh+International+Joint+Conference+on)
35. Grönroos, S.-A., Virpioja, S., and Kurimo, M. (2015). Tuning Phrase-Based Segmented Translation for a Morphologically Complex Target Language. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 105–111, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gro%CC%88nroos%2C+S%2E%2DA%2E%2C+Virpioja%2C+S%2E%2C+and+Kurimo%2C+M%2E+%282015%29%2E+Tuning+Phrase%2DBased+Segmented+Translation+for+a+Morphologically+Complex+Target+Language%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Mach)
36. Gwinnup, J., Anderson, T., Erdmann, G., Young, K., May, C., Kazi, M., Salesky, E., and Thompson, B. (2015). The AFRL-MITLL WMT15 System: Theres More than One Way to Decode It! In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 112– 119, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gwinnup%2C+J%2E%2C+Anderson%2C+T%2E%2C+Erdmann%2C+G%2E%2C+Young%2C+K%2E%2C+May%2C+C%2E%2C+Kazi%2C+M%2E%2C+Salesky%2C+E%2E%2C+and+Thompson%2C+B%2E+%282015%29%2E+The+AFRL%2DMITLL+WMT15+System%3A+Theres+More+than+One+Way+to+Decode+It%21+In+Proceedings+of+the+Te)
37. Ha, T.-L., Do, Q.-K., Cho, E., Niehues, J., Allauzen, A., Yvon, F., and Waibel, A. (2015). The KIT-LIMSI Translation System for WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 120– 125, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ha%2C+T%2E%2DL%2E%2C+Do%2C+Q%2E%2DK%2E%2C+Cho%2C+E%2E%2C+Niehues%2C+J%2E%2C+Allauzen%2C+A%2E%2C+Yvon%2C+F%2E%2C+and+Waibel%2C+A%2E+%282015%29%2E+The+KIT%2DLIMSI+Translation+System+for+WMT+2015%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Tr)
38. Haddow, B., Huck, M., Birch, A., Bogoychev, N., and Koehn, P. (2015). The Edinburgh/JHU Phrase-based Machine Translation Systems for WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 126–133, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Haddow%2C+B%2E%2C+Huck%2C+M%2E%2C+Birch%2C+A%2E%2C+Bogoychev%2C+N%2E%2C+and+Koehn%2C+P%2E+%282015%29%2E+The+Edinburgh%2FJHU+Phrase%2Dbased+Machine+Translation+Systems+for+WMT+2015%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machi)
39. Heafield, K. (2011). KenLM: faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, United Kingdom. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Heafield%2C+K%2E+%282011%29%2E+KenLM%3A+faster+and+smaller+language+model+queries%2E+In+Proceedings+of+the+Sixth+Workshop+on+Statistical+Machine+Translation%2C+pages+187%E2%80%93197%2C+Edinburgh%2C+United+Kingdom%2E+Association+fo) [our card](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/)
40. Jean, S., Firat, O., Cho, K., Memisevic, R., and Bengio, Y. (2015). Montreal Neural Machine Translation Systems for WMT15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jean%2C+S%2E%2C+Firat%2C+O%2E%2C+Cho%2C+K%2E%2C+Memisevic%2C+R%2E%2C+and+Bengio%2C+Y%2E+%282015%29%2E+Montreal+Neural+Machine+Translation+Systems+for+WMT15%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+page)
41. Junczys-Dowmunt, M. and Szal, A. (2011). SyM-Giza++: Symmetrized Word Alignment Models for Statistical Machine Translation. In SIIS, volume 7053 of Lecture Notes in Computer Science, pages 379–390. Springer. [find](https://scholar.google.com/scholar?q=Junczys%2DDowmunt%2C+M%2E+and+Szal%2C+A%2E+%282011%29%2E+SyM%2DGiza%2B%2B%3A+Symmetrized+Word+Alignment+Models+for+Statistical+Machine+Translation%2E+In+SIIS%2C+volume+7053+of+Lecture+Notes+in+Computer+Science%2C+pages+379%E2%80%93390%2E+Sp)
42. Koehn, P. (2004). Statistical Significance Tests for Machine Translation Evaluation. In Lin, D. and Wu, D., editors, Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E+%282004%29%2E+Statistical+Significance+Tests+for+Machine+Translation+Evaluation%2E+In+Lin%2C+D%2E+and+Wu%2C+D%2E%2C+editors%2C+Proceedings+of+EMNLP+2004%2C+pages+388%E2%80%93395%2C+Barcelona%2C+Spain%2E)
43. Koehn, P. (2005). Europarl: A Parallel Corpus for Statistical Machine Translation. In MT Summit X. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E+%282005%29%2E+Europarl%3A+A+Parallel+Corpus+for+Statistical+Machine+Translation%2E+In+MT+Summit+X%2E)
44. Koehn, P. (2010). Statistical Machine Translation. Cambridge University Press, New York, NY, USA, 1st edition. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E+%282010%29%2E+Statistical+Machine+Translation%2E+Cambridge+University+Press%2C+New+York%2C+NY%2C+USA%2C+1st+edition%2E)
45. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In ACL 2007 Demonstrations, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E%2C+Hoang%2C+H%2E%2C+Birch%2C+A%2E%2C+Callison%2DBurch%2C+C%2E%2C+Federico%2C+M%2E%2C+Bertoldi%2C+N%2E%2C+Cowan%2C+B%2E%2C+Shen%2C+W%2E%2C+Moran%2C+C%2E%2C+Zens%2C+R%2E%2C+Dyer%2C+C%2E%2C+Bojar%2C+O%2E%2C+Constantin%2C+A%2E%2C+and+Herbst%2C+E%2E+%282007%29%2E+Moses%3A+Open+sourc)
46. Koehn, P. and Monz, C. (2006). Manual and automatic evaluation of machine translation between European languages. In Proceedings of NAACL 2006 Workshop on Statistical Machine Translation, New York, New York. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E+and+Monz%2C+C%2E+%282006%29%2E+Manual+and+automatic+evaluation+of+machine+translation+between+European+languages%2E+In+Proceedings+of+NAACL+2006+Workshop+on+Statistical+Machine+Translation%2C+New+York%2C+Ne) [our card](https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/)
47. Kolachina, P. and Ranta, A. (2015). GF Wide-coverage English-Finnish MT system for WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 141– 144, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kolachina%2C+P%2E+and+Ranta%2C+A%2E+%282015%29%2E+GF+Wide%2Dcoverage+English%2DFinnish+MT+system+for+WMT+2015%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+141%E2%80%93+144%2C+Lisboa%2C+Portugal%2E+)
48. Kreutzer, J., Schamoni, S., and Riezler, S. (2015). QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 316–322, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kreutzer%2C+J%2E%2C+Schamoni%2C+S%2E%2C+and+Riezler%2C+S%2E+%282015%29%2E+QUality+Estimation+from+ScraTCH+%28QUETCH%29%3A+Deep+Learning+for+Word%2Dlevel+Translation+Quality+Estimation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Stati)
49. Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33:159–174. [find](https://scholar.google.com/scholar?q=Landis%2C+J%2E+R%2E+and+Koch%2C+G%2E+G%2E+%281977%29%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+Biometrics%2C+33%3A159%E2%80%93174%2E)
50. Langlois, D. (2015). LORIA System for the WMT15 Quality Estimation Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 323–329, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Langlois%2C+D%2E+%282015%29%2E+LORIA+System+for+the+WMT15+Quality+Estimation+Shared+Task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+323%E2%80%93329%2C+Lisboa%2C+Portugal%2E+Association+fo)
51. Lavie, A. and Agarwal, A. (2007). Meteor: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 228–231. [find](https://scholar.google.com/scholar?q=Lavie%2C+A%2E+and+Agarwal%2C+A%2E+%282007%29%2E+Meteor%3A+An+Automatic+Metric+for+MT+Evaluation+with+High+Levels+of+Correlation+with+Human+Judgments%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+Trans) [our card](https://md.research.elamur.ai/free/en/papers/lavie-2007-meteor-automatic-metric-mt-evaluation/)
52. Liang, P., Taskar, B., and Klein, D. (2006). Alignment by Agreement. In HLTNAACL, New York. [find](https://scholar.google.com/scholar?q=Liang%2C+P%2E%2C+Taskar%2C+B%2E%2C+and+Klein%2C+D%2E+%282006%29%2E+Alignment+by+Agreement%2E+In+HLTNAACL%2C+New+York%2E) [our card](https://md.research.elamur.ai/free/en/papers/liang-2006-alignment-agreement/)
53. Logacheva, V., Hokamp, C., and Specia, L. (2015). Data enhancement and selection strategies for the word-level Quality Estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 330–335, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Logacheva%2C+V%2E%2C+Hokamp%2C+C%2E%2C+and+Specia%2C+L%2E+%282015%29%2E+Data+enhancement+and+selection+strategies+for+the+word%2Dlevel+Quality+Estimation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translati)
54. Luong, N. Q., Besacier, L., and Lecouteux, B. (2014). LIG System for Word Level QE task at WMT14. In Ninth Workshop on Statistical Machine Translation, pages 335–341, Baltimore, Maryland, USA. Association for Computational Linguistics. [↗](https://doi.org/10.13140/rg.2.1.3971.3446)
55. Luong, N. Q., Lecouteux, B., and Besacier, L. (2013). LIG system for WMT13 QE task: Investigating the usefulness of features in word confidence estimation for MT. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 384–389, Sofia, Bulgaria. Association for Computational Linguistics. [↗](https://doi.org/10.13140/rg.2.1.4757.7764)
56. Marie, B., Allauzen, A., Burlot, F., Do, Q.-K., Ive, J., knyazeva, e., Labeau, M., Lavergne, T., Löser, K., Pécheux, N., and Yvon, F. (2015). LIMSI@WMT’15 : Translation Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 145–151, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marie%2C+B%2E%2C+Allauzen%2C+A%2E%2C+Burlot%2C+F%2E%2C+Do%2C+Q%2E%2DK%2E%2C+Ive%2C+J%2E%2C+knyazeva%2C+e%2E%2C+Labeau%2C+M%2E%2C+Lavergne%2C+T%2E%2C+Lo%CC%88ser%2C+K%2E%2C+Pe%CC%81cheux%2C+N%2E%2C+and+Yvon%2C+F%2E+%282015%29%2E+LIMSI%40WMT%E2%80%9915+%3A+Translation+Task%2E+In+Proceedings+of+the+T)
57. Och, F. J. (2003). Minimum Error Rate Training in Statistical Machine Translation. In ACL03, pages 160–167, Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Och%2C+F%2E+J%2E+%282003%29%2E+Minimum+Error+Rate+Training+in+Statistical+Machine+Translation%2E+In+ACL03%2C+pages+160%E2%80%93167%2C+Sapporo%2C+Japan%2E)
58. Padr, L. and Stanilovsky, E. (2012). Freeling 3.0: Towards wider multilinguality. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. [find](https://scholar.google.com/scholar?q=Padr%2C+L%2E+and+Stanilovsky%2C+E%2E+%282012%29%2E+Freeling+3%2E0%3A+Towards+wider+multilinguality%2E+In+Proceedings+of+the+Eight+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+Istanbul%2C+Turkey%2E) [our card](https://md.research.elamur.ai/free/en/papers/padro-2012-freeling-3-0-wider-multilinguality/)
59. Pal, S., Naskar, S., and Bandyopadhyay, S. (2013). A Hybrid Word Alignment Model for Phrase-Based Statistical Machine Translation. In Proceedings of the Second Workshop on Hybrid Approaches to Translation, pages 94–101, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Pal%2C+S%2E%2C+Naskar%2C+S%2E%2C+and+Bandyopadhyay%2C+S%2E+%282013%29%2E+A+Hybrid+Word+Alignment+Model+for+Phrase%2DBased+Statistical+Machine+Translation%2E+In+Proceedings+of+the+Second+Workshop+on+Hybrid+Approaches+to+Transla)
60. Pal, S., Naskar, S., and van Genabith, J. (2015a). UdS-Sant: English–German Hybrid Machine Translation System. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 152–157, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pal%2C+S%2E%2C+Naskar%2C+S%2E%2C+and+van+Genabith%2C+J%2E+%282015a%29%2E+UdS%2DSant%3A+English%E2%80%93German+Hybrid+Machine+Translation+System%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+152%E2%80%93157%2C+L)
61. Pal, S., Vela, M., Naskar, S. K., and van Genabith, J. (2015b). USAAR-SAPE: An English– Spanish Statistical Automatic Post-Editing System. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 216– 221, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pal%2C+S%2E%2C+Vela%2C+M%2E%2C+Naskar%2C+S%2E+K%2E%2C+and+van+Genabith%2C+J%2E+%282015b%29%2E+USAAR%2DSAPE%3A+An+English%E2%80%93+Spanish+Statistical+Automatic+Post%2DEditing+System%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+T)
62. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Morristown, NJ, USA. [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+Roukos%2C+S%2E%2C+Ward%2C+T%2E%2C+and+Zhu%2C+W%2E%2DJ%2E+%282002%29%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+on+Association+for+Computational+Li) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
63. Parton, K., Habash, N., McKeown, K., Iglesias, G., and de Gispert, A. (2012). Can Automatic Post-Editing Make MT More Meaningful? In Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), pages 111–118, Trento, Italy. [↗](https://doi.org/10.7916/d80v8n4b)
64. Peter, J.-T., Toutounchi, F., Wuebker, J., and Ney, H. (2015). The RWTH Aachen German-English Machine Translation System for WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 158–163, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peter%2C+J%2E%2DT%2E%2C+Toutounchi%2C+F%2E%2C+Wuebker%2C+J%2E%2C+and+Ney%2C+H%2E+%282015%29%2E+The+RWTH+Aachen+German%2DEnglish+Machine+Translation+System+for+WMT+2015%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Trans)
65. Potet, M., Esperana-Rodier, E., Besacier, L., and Blanchon, H. (2012). Collection of a large database of french-english smt output corrections. In LREC, pages 4043–4048. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Potet%2C+M%2E%2C+Esperana%2DRodier%2C+E%2E%2C+Besacier%2C+L%2E%2C+and+Blanchon%2C+H%2E+%282012%29%2E+Collection+of+a+large+database+of+french%2Denglish+smt+output+corrections%2E+In+LREC%2C+pages+4043%E2%80%934048%2E+European+Language+Resources+As)
66. Quernheim, D. (2015). Exact Decoding with Multi Bottom-Up Tree Transducers. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 164–171, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Quernheim%2C+D%2E+%282015%29%2E+Exact+Decoding+with+Multi+Bottom%2DUp+Tree+Transducers%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+164%E2%80%93171%2C+Lisboa%2C+Portugal%2E+Association+for+Co)
67. Raybaud, S., Langlois, D., and Smali, K. (2011). this sentence is wrong. detecting errors in machine-translated sentences. Machine Translation, 25(1):1–34. [↗](https://doi.org/10.1007/s10590-011-9094-9)
68. Rubino, R., Pirinen, T., Esplà-Gomis, M., Ljubešić, N., Ortiz Rojas, S., Papavassiliou, V., Prokopidis, P., and Toral, A. (2015). Abu-MaTran at WMT 2015 Translation Task: Morphological Segmentation and Web Crawling. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 184–191, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rubino%2C+R%2E%2C+Pirinen%2C+T%2E%2C+Espla%CC%80%2DGomis%2C+M%2E%2C+Ljubes%CC%8Cic%CC%81%2C+N%2E%2C+Ortiz+Rojas%2C+S%2E%2C+Papavassiliou%2C+V%2E%2C+Prokopidis%2C+P%2E%2C+and+Toral%2C+A%2E+%282015%29%2E+Abu%2DMaTran+at+WMT+2015+Translation+Task%3A+Morphological+Segmentation)
69. Scarton, C., Tan, L., and Specia, L. (2015a). USHEF and USAAR-USHEF participation in the WMT15 QE shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 336–341, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Scarton%2C+C%2E%2C+Tan%2C+L%2E%2C+and+Specia%2C+L%2E+%282015a%29%2E+USHEF+and+USAAR%2DUSHEF+participation+in+the+WMT15+QE+shared+task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+336%E2%80%93341%2C+L)
70. Scarton, C., Zampieri, M., Vela, M., van Genabith, J., and Specia, L. (2015b). Searching for Context: a Study on Document-Level Labels for Translation Quality Estimation. In The 18th Annual Conference of the European Association for Machine Translation, pages 121–128, Antalya, Turkey. [find](https://scholar.google.com/scholar?q=Scarton%2C+C%2E%2C+Zampieri%2C+M%2E%2C+Vela%2C+M%2E%2C+van+Genabith%2C+J%2E%2C+and+Specia%2C+L%2E+%282015b%29%2E+Searching+for+Context%3A+a+Study+on+Document%2DLevel+Labels+for+Translation+Quality+Estimation%2E+In+The+18th+Annual+Conference)
71. Schwartz, L., Bryce, B., Geigle, C., Massung, S., Liu, Y., Peng, H., Raja, V., Roy, S., and Upadhyay, S. (2015). The University of Illinois submission to the WMT 2015 Shared Translation Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 192– 198, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Schwartz%2C+L%2E%2C+Bryce%2C+B%2E%2C+Geigle%2C+C%2E%2C+Massung%2C+S%2E%2C+Liu%2C+Y%2E%2C+Peng%2C+H%2E%2C+Raja%2C+V%2E%2C+Roy%2C+S%2E%2C+and+Upadhyay%2C+S%2E+%282015%29%2E+The+University+of+Illinois+submission+to+the+WMT+2015+Shared+Translation+Task%2E+In+Proce)
72. Shah, K., Logacheva, V., Paetzold, G., Blain, F., Beck, D., Bougares, F., and Specia, L. (2015). SHEF-NN: Translation Quality Estimation with Neural Networks. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 342–347, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shah%2C+K%2E%2C+Logacheva%2C+V%2E%2C+Paetzold%2C+G%2E%2C+Blain%2C+F%2E%2C+Beck%2C+D%2E%2C+Bougares%2C+F%2E%2C+and+Specia%2C+L%2E+%282015%29%2E+SHEF%2DNN%3A+Translation+Quality+Estimation+with+Neural+Networks%2E+In+Proceedings+of+the+Tenth+Workshop+on+S)
73. Shang, L., Cai, D., and Ji, D. (2015). Strategy-Based Technology for Estimating MT Quality. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 348–352, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shang%2C+L%2E%2C+Cai%2C+D%2E%2C+and+Ji%2C+D%2E+%282015%29%2E+Strategy%2DBased+Technology+for+Estimating+MT+Quality%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+348%E2%80%93352%2C+Lisboa%2C+Portugal%2E+As)
74. Simard, M., Goutte, C., and Isabelle, P. (2007). Statistical Phrase-Based Post-Editing. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT), pages 508–515, Rochester, New York. [find](https://scholar.google.com/scholar?q=Simard%2C+M%2E%2C+Goutte%2C+C%2E%2C+and+Isabelle%2C+P%2E+%282007%29%2E+Statistical+Phrase%2DBased+Post%2DEditing%2E+In+Proceedings+of+the+Annual+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Lingu) [our card](https://md.research.elamur.ai/free/en/papers/simard-2007-statistical-phrase-based-post-editing/)
75. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006a). A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas, pages 223– 231, Cambridge, Massachusetts, USA. [find](https://scholar.google.com/scholar?q=Snover%2C+M%2E%2C+Dorr%2C+B%2E%2C+Schwartz%2C+R%2E%2C+Micciulla%2C+L%2E%2C+and+Makhoul%2C+J%2E+%282006a%29%2E+A+Study+of+Translation+Edit+Rate+with+Targeted+Human+Annotation%2E+In+Proceedings+of+Association+for+Machine+Translation+in+th) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
76. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006b). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA-2006), Cambridge, Massachusetts. [find](https://scholar.google.com/scholar?q=Snover%2C+M%2E%2C+Dorr%2C+B%2E%2C+Schwartz%2C+R%2E%2C+Micciulla%2C+L%2E%2C+and+Makhoul%2C+J%2E+%282006b%29%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proceedings+of+the+7th+Biennial+Conference+of+the+Associ) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
77. Specia, L., Paetzold, G., and Scarton, C. (2015). Multi-level Translation Quality Prediction with QuEst++. In 53rd Annual Meeting of the Association for Computational Linguistics and Seventh International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing: System Demonstrations, pages 115–120, Beijing, China. [find](https://scholar.google.com/scholar?q=Specia%2C+L%2E%2C+Paetzold%2C+G%2E%2C+and+Scarton%2C+C%2E+%282015%29%2E+Multi%2Dlevel+Translation+Quality+Prediction+with+QuEst%2B%2B%2E+In+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+Seventh+Internati) [our card](https://md.research.elamur.ai/free/en/papers/specia-2015-multi-level-translation-quality-prediction/)
78. Specia, L., Shah, K., de Souza, J. G., and Cohn, T. (2013). QuEst - A translation quality estimation framework. In 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL-2013, pages 79– 84, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Specia%2C+L%2E%2C+Shah%2C+K%2E%2C+de+Souza%2C+J%2E+G%2E%2C+and+Cohn%2C+T%2E+%282013%29%2E+QuEst+%2D+A+translation+quality+estimation+framework%2E+In+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+System+Demonstr) [our card](https://md.research.elamur.ai/free/en/papers/specia-2013-quest-translation-quality-estimation-framework/)
79. Stanojević, M., Kamran, A., and Bojar, O. (2015a). Results of the WMT15 Tuning Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 274– 281, Lisboa, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18154/rwth-2016-07012)
80. Stanojević, M., Kamran, A., Koehn, P., and Bojar, O. (2015b). Results of the WMT15 Metrics Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256–273, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Stanojevic%CC%81%2C+M%2E%2C+Kamran%2C+A%2E%2C+Koehn%2C+P%2E%2C+and+Bojar%2C+O%2E+%282015b%29%2E+Results+of+the+WMT15+Metrics+Shared+Task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+256%E2%80%93273%2C+Lisboa%2C)
81. Steele, D., Sim Smith, K., and Specia, L. (2015). Sheffield Systems for the Finnish-English WMT Translation Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 172–176, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Steele%2C+D%2E%2C+Sim+Smith%2C+K%2E%2C+and+Specia%2C+L%2E+%282015%29%2E+Sheffield+Systems+for+the+Finnish%2DEnglish+WMT+Translation+Task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+172%E2%80%93176)
82. Tezcan, A., Hoste, V., Desmet, B., and Macken, L. (2015). UGENT-LT3 SCATE System for Machine Translation Quality Estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 353–360, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tezcan%2C+A%2E%2C+Hoste%2C+V%2E%2C+Desmet%2C+B%2E%2C+and+Macken%2C+L%2E+%282015%29%2E+UGENT%2DLT3+SCATE+System+for+Machine+Translation+Quality+Estimation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pa)
83. Tiedemann, J., Ginter, F., and Kanerva, J. (2015). Morphological Segmentation and OPUS for Finnish-English Machine Translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 177–183, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tiedemann%2C+J%2E%2C+Ginter%2C+F%2E%2C+and+Kanerva%2C+J%2E+%282015%29%2E+Morphological+Segmentation+and+OPUS+for+Finnish%2DEnglish+Machine+Translation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C)
84. Williams, P., Sennrich, R., Nadejde, M., Huck, M., and Koehn, P. (2015). Edinburgh’s Syntax-Based Systems at WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 199–209, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Williams%2C+P%2E%2C+Sennrich%2C+R%2E%2C+Nadejde%2C+M%2E%2C+Huck%2C+M%2E%2C+and+Koehn%2C+P%2E+%282015%29%2E+Edinburgh%E2%80%99s+Syntax%2DBased+Systems+at+WMT+2015%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+19)
85. Wisniewski, G., Pécheux, N., and Yvon, F. (2015). Why Predicting Post-Edition is so Hard? Failure Analysis of LIMSI Submission to the APE Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 222–227, Lisboa, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wisniewski%2C+G%2E%2C+Pe%CC%81cheux%2C+N%2E%2C+and+Yvon%2C+F%2E+%282015%29%2E+Why+Predicting+Post%2DEdition+is+so+Hard%3F+Failure+Analysis+of+LIMSI+Submission+to+the+APE+Shared+Task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statisti)
86. Wisniewski, G., Singh, A. K., Segal, N., and Yvon, F. (2013). Design and Analysis of a Large Corpus of Post-edited Translations: Quality Estimation, Failure Analysis and the Variability of Post-edition. Machine Translation Summit, 14:117–124. [find](https://scholar.google.com/scholar?q=Wisniewski%2C+G%2E%2C+Singh%2C+A%2E+K%2E%2C+Segal%2C+N%2E%2C+and+Yvon%2C+F%2E+%282013%29%2E+Design+and+Analysis+of+a+Large+Corpus+of+Post%2Dedited+Translations%3A+Quality+Estimation%2C+Failure+Analysis+and+the+Variability+of+Post%2Deditio)