---
title: "Neural versus Phrase-Based Machine — Bentivogli 2016"
description: "In 2015, neural MT clearly beat strong phrase-based systems on English-German at IWSLT."
authors: ["Luisa Bentivogli", "Arianna Bisazza", "Mauro Cettolo", "Marcello Federico"]
year: 2016
venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D16-1025/"
doi: "10.18653/v1/d16-1025"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bentivogli-2016-neural-versus-phrase-based-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bentivogli-2016-neural-versus-phrase-based-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Neural versus Phrase-Based Machine Translation Quality: a Case Study

# Neural versus Phrase-Based Machine Translation Quality: a Case Study

**Authors:** [Luisa Bentivogli](https://md.research.elamur.ai/free/en/authors/luisa-bentivogli/), [Arianna Bisazza](https://md.research.elamur.ai/free/en/authors/arianna-bisazza/), [Mauro Cettolo](https://md.research.elamur.ai/free/en/authors/mauro-cettolo/), [Marcello Federico](https://md.research.elamur.ai/free/en/authors/marcello-federico/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D16-1025/) · [PDF](https://aclanthology.org/D16-1025.pdf) · [DOI 10.18653/v1/d16-1025](https://doi.org/10.18653/v1/d16-1025) · [OpenAlex](https://openalex.org/W2513073850)

**Citations:** 316 citations (openalex-snapshot · 2026-10-14) · [120](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
In 2015, neural MT clearly beat strong phrase-based systems on English-German at IWSLT. BLEU showed the size of the gain but not its source. This study asks which linguistic aspects of translation the neural approach handles better, and which it still gets wrong.

The authors compared four top systems from the IWSLT 2015 English-German TED-talk task:
- one attention-based LSTM NMT system;
- three phrase-based variants: standard, hierarchical, and phrase+syntax combination.

Professional translators made minimal post-edits of each system's output. The authors used these post-edits with TER-based alignment to measure post-editing effort and to automatically sort errors into three types: morphological, lexical and word-order. They then examined how quality changes with sentence length and talk content, and analysed reordering errors by linguistic constituent.

NMT needed about a quarter less post-editing than the best phrase-based system and made fewer errors of every type. Its biggest gain was in placing verbs. It was weaker on long sentences and on reorderings that depend on meaning. The work matters because it gave the first diagnostic, error-level picture of NMT's strengths.

## Key points
- **Data:** IWSLT 2015 English-German (TED transcripts, available via WIT3). The Human Evaluation set covers the first half of each of 12 test talks, about 600 sentences and roughly 10K words.
- **Post-editing setup:** Five professional translators each post-edited 120 sentences per system. This produced five new references per sentence, one targeted and four extra for each system.
- **Systems compared:** PBSY, HPB, SPB, and NMT (Luong & Manning, 2015). The NMT system was an ensemble of 8 four-layer LSTMs with attention, source reversing and 50K vocabularies. It was the only system trained without monolingual data.
- **Overall results for NMT versus the strongest competitor:**
  - about 26% less post-editing effort (HTER);
  - 19% fewer morphology errors;
  - 17% fewer lexical errors;
  - about 50% fewer word-order errors;
  - about 70% fewer verb-placement errors.
- **Remaining weaknesses:** NMT led at all sentence lengths, but its quality fell faster as input length grew. It did no better than phrase-based systems at placing the negation particle *nicht* or at ordering adjunct prepositional phrases.

## Interesting ideas
- **Post-edits as targeted references.** Minimal professional post-edits act as references tailored to each output. This gives more reliable error signals than one independent reference, and the idea could be reused for evaluating any generation system.
- **Cheap error typing from TER alignments.** TER shifts and substitutions on post-edits can be mapped to simple error classes such as word order versus lexical choice. This sits between costly manual annotation and noisy reference-based analysis. It also works for opaque models, because it needs no phrase table or other internal resources.
- **Balanced annotation design.** Every translator post-edited an equal share of every system's output. This keeps annotator differences from biasing the system comparison, which is a useful rule for any multi-system human evaluation.
- **Breaking errors down by constituent.** Reordering errors were grouped by syntactic role (auxiliary-main verb, verb in subordinate clause, prepositional phrase, negation). This shows where a model improves rather than only how much, and the approach could transfer to other languages with a parser.
- **Watching for metric artefacts.** The authors noticed that HTER sometimes labels frequent words like articles as shifts when they are really unrelated substitutions. Checking automatic error categories against real examples in this way is a good habit before trusting the counts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 109 not yet.

- [ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/amrhein-2022-aces-translation-accuracy-challenge-sets/)
- [What do Neural Machine Translation Models Learn about Morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/)
- [Findings of the WMT 2018 Shared Task on Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/chatterjee-2018-findings-wmt-2018-shared-task/)
- [Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings](https://md.research.elamur.ai/free/en/papers/gonzales-2017-improving-word-sense-disambiguation-neural/)
- [A Challenge Set Approach to Evaluating Machine Translation](https://md.research.elamur.ai/free/en/papers/isabelle-2017-challenge-set-approach-evaluating-machine/)
- [Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets](https://md.research.elamur.ai/free/en/papers/moghe-2025-machine-translation-meta-evaluation-through/)
- [Analyzing Uncertainty in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/ott-2018-analyzing-uncertainty-neural-machine-translation/)
- [Investigating Backtranslation in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/)
- [Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/)
- [Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation](https://md.research.elamur.ai/free/en/papers/vanmassenhove-2019-lost-translation-loss-decay-linguistic/)
- [Handling Syntactic Divergence in Low-resource Machine Translation](https://md.research.elamur.ai/free/en/papers/zhou-2019-handling-syntactic-divergence-low-resource/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bentivogli2016neural,
  title = {{Neural versus Phrase-Based Machine Translation Quality: a Case Study}},
  author = {Bentivogli, Luisa and Bisazza, Arianna and Cettolo, Mauro and Federico, Marcello},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1025},
  url = {https://doi.org/10.18653/v1/d16-1025}
}
```

- [← previous Neural Text Generation from Structured Data with Application to the Biography Domain](https://md.research.elamur.ai/free/en/papers/lebret-2016-neural-text-generation-structured-data/)
- [next → OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles](https://md.research.elamur.ai/free/en/papers/lison-2016-opensubtitles2016-extracting-large-parallel-corpora/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR, San Diego, US-CA. [↗](https://doi.org/10.48550/arxiv.1409.0473) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Alexandra Birch, Miles Osborne, and Phil Blunsom. 2010. Metrics for MT evaluation: evaluating reordering. Machine Translation, 24(1):15–26. [↗](https://doi.org/10.1007/s10590-009-9066-5)
3. Alexandra Birch. 2011. Reordering Metrics for Statistical Machine Translation. Ph.D. thesis, School of Informatics, University of Edinburgh, UK. [find](https://scholar.google.com/scholar?q=Alexandra+Birch%2E+2011%2E+Reordering+Metrics+for+Statistical+Machine+Translation%2E+Ph%2ED%2E+thesis%2C+School+of+Informatics%2C+University+of+Edinburgh%2C+UK%2E)
4. Arianna Bisazza and Marcello Federico. 2013. Efficient solutions for word reordering in German-English phrase-based statistical machine translation. In Proc. of WMT, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Arianna+Bisazza+and+Marcello+Federico%2E+2013%2E+Efficient+solutions+for+word+reordering+in+German%2DEnglish+phrase%2Dbased+statistical+machine+translation%2E+In+Proc%2E+of+WMT%2C+Sofia%2C+Bulgaria%2E)
5. Eduardo Blanco and Dan Moldovan. 2011. Semantic representation of negation using focus detection. In Proc. of ACL-HLT, Portland, US-OR. [find](https://scholar.google.com/scholar?q=Eduardo+Blanco+and+Dan+Moldovan%2E+2011%2E+Semantic+representation+of+negation+using+focus+detection%2E+In+Proc%2E+of+ACL%2DHLT%2C+Portland%2C+US%2DOR%2E)
6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proc. of WMT, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Barry+Haddow%2C+Matthias+Huck%2C+Chris+Hokamp%2C+Philipp+Koehn%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Matt+Post%2C+Carolina+Scarton%2C+Lucia+Speci) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/)
7. Ondrej Bojar. 2011. Analyzing error types in English-Czech machine translation. The Prague Bulletin of Mathematical Linguistic, (95):63–76. [↗](https://doi.org/10.2478/v10108-011-0005-2)
8. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web Inventory of Transcribed and Translated Talks. In Proc. of EAMT, Trento, Italy. [find](https://scholar.google.com/scholar?q=Mauro+Cettolo%2C+Christian+Girardi%2C+and+Marcello+Federico%2E+2012%2E+WIT+3+%3A+Web+Inventory+of+Transcribed+and+Translated+Talks%2E+In+Proc%2E+of+EAMT%2C+Trento%2C+Italy%2E)
9. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015. [find](https://scholar.google.com/scholar?q=Mauro+Cettolo%2C+Jan+Niehues%2C+Sebastian+Stu%CC%88ker%2C+Luisa+Bentivogli%2C+Roldano+Cattoni%2C+and+Marcello+Federico%2E+2015%2E)
10. The IWSLT 2015 evaluation campaign. In Proc. of IWSLT, Da Nang, Vietnam. [find](https://scholar.google.com/scholar?q=The+IWSLT+2015+evaluation+campaign%2E+In+Proc%2E+of+IWSLT%2C+Da+Nang%2C+Vietnam%2E)
11. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: encoder–decoder approaches. In Proc. of SSST-8, Doha, Qatar. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrie%CC%88nboer%2C+Dzmitry+Bahdanau%2C+and+Yoshua+Bengio%2E+2014a%2E+On+the+properties+of+neural+machine+translation%3A+encoder%E2%80%93decoder+approaches%2E+In+Proc%2E+of+SSST%2D8%2C+Doha%2C+Qatar%2E) [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
12. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proc. of EMNLP, Doha, Qatar. [↗](https://doi.org/10.48550/arxiv.1406.1078)
13. Joke Daems, Lieve Macken, and Sonia Vandepitte. 2014. On the origin of errors: a fine-grained analysis of MT and PE errors and their relationship. In Proc. of LREC, Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=Joke+Daems%2C+Lieve+Macken%2C+and+Sonia+Vandepitte%2E+2014%2E+On+the+origin+of+errors%3A+a+fine%2Dgrained+analysis+of+MT+and+PE+errors+and+their+relationship%2E+In+Proc%2E+of+LREC%2C+Reykjavik%2C+Iceland%2E)
14. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proc. of NACL-HLT, Atlanta, US-GA. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A%2E+Smith%2E+2013%2E+A+simple%2C+fast%2C+and+effective+reparameterization+of+IBM+model+2%2E+In+Proc%2E+of+NACL%2DHLT%2C+Atlanta%2C+US%2DGA%2E)
15. Mireia Farrús Cabeceran, Marta Ruiz Costa-Jussà, José Bernardo Mariño Acebal, and José Adrián Rodrı́guez Fonollosa. 2010. Linguistic-based evaluation criteria to identify statistical machine translation errors. In Proc. of EAMT, Saint-Raphaël, France. [find](https://scholar.google.com/scholar?q=Mireia+Farru%CC%81s+Cabeceran%2C+Marta+Ruiz+Costa%2DJussa%CC%80%2C+Jose%CC%81+Bernardo+Marin%CC%83o+Acebal%2C+and+Jose%CC%81+Adria%CC%81n+Rodr%C4%B1%CC%81guez+Fonollosa%2E+2010%2E+Linguistic%2Dbased+evaluation+criteria+to+identify+statistical+machine+tra)
16. Marcello Federico, Matteo Negri, Luisa Bentivogli, and Marco Turchi. 2014. Assessing the impact of translation errors on machine translation quality with mixed-effects models. In Proc. of EMNLP, Doha, Qatar. [find](https://scholar.google.com/scholar?q=Marcello+Federico%2C+Matteo+Negri%2C+Luisa+Bentivogli%2C+and+Marco+Turchi%2E+2014%2E+Assessing+the+impact+of+translation+errors+on+machine+translation+quality+with+mixed%2Deffects+models%2E+In+Proc%2E+of+EMNLP%2C+Doha%2C)
17. Mark Fishel, Ondrej Bojar, and Maja Popović. 2012. Terra: a collection of translation error-annotated corpora. In Proc. of LREC, Istanbul, Turkey. [find](https://scholar.google.com/scholar?q=Mark+Fishel%2C+Ondrej+Bojar%2C+and+Maja+Popovic%CC%81%2E+2012%2E+Terra%3A+a+collection+of+translation+error%2Dannotated+corpora%2E+In+Proc%2E+of+LREC%2C+Istanbul%2C+Turkey%2E)
18. Mary Flanagan. 1994. Error classification for MT evaluation. In Proc. of AMTA, Columbia, US-MD. [find](https://scholar.google.com/scholar?q=Mary+Flanagan%2E+1994%2E+Error+classification+for+MT+evaluation%2E+In+Proc%2E+of+AMTA%2C+Columbia%2C+US%2DMD%2E)
19. Thomas François and Cédrick Fairon. 2012. An “AI readability” formula for French as a foreign language. In Proc. of EMNLP-CoNLL, Jeju Island, Korea. [find](https://scholar.google.com/scholar?q=Thomas+Franc%CC%A7ois+and+Ce%CC%81drick+Fairon%2E+2012%2E+An+%E2%80%9CAI+readability%E2%80%9D+formula+for+French+as+a+foreign+language%2E+In+Proc%2E+of+EMNLP%2DCoNLL%2C+Jeju+Island%2C+Korea%2E)
20. Çaglar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loı̈c Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. CoRR, abs/1503.03535. [↗](https://arxiv.org/abs/1503.03535) [our card](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/)
21. Thanh-Le Ha, Jan Niehues, Eunah Cho, Mohammed Mediani, and Alex Waibel. 2015. The KIT translation systems for IWSLT 2015. In Proc. of IWSLT, Da Nang, Vietnam. [↗](https://doi.org/10.5445/ir/1000051636)
22. Matthias Huck and Alexandra Birch. 2015. The Edinburgh machine translation systems for IWSLT 2015. In Proc. of IWSLT, Da Nang, Vietnam. [find](https://scholar.google.com/scholar?q=Matthias+Huck+and+Alexandra+Birch%2E+2015%2E+The+Edinburgh+machine+translation+systems+for+IWSLT+2015%2E+In+Proc%2E+of+IWSLT%2C+Da+Nang%2C+Vietnam%2E)
23. Ann Irvine, John Morgan, Marine Carpuat, Hal Daumé III, and Dragos Munteanu. 2013. Measuring machine translation errors in new domains. Transactions of the Association for Computational Linguistics, 1:429– 440. [↗](https://doi.org/10.1162/tacl_a_00239) [our card](https://md.research.elamur.ai/free/en/papers/irvine-2013-measuring-machine-translation-errors-new/)
24. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015a. On using very large target vocabulary for neural machine translation. In Proc. of ACL-IJCNLP, Beijing, China. [↗](https://doi.org/10.48550/arxiv.1412.2007) [our card](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
25. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015b. Montreal neural machine translation systems for WMT15. In Proc. of WMT, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Se%CC%81bastien+Jean%2C+Orhan+Firat%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2015b%2E+Montreal+neural+machine+translation+systems+for+WMT15%2E+In+Proc%2E+of+WMT%2C+Lisbon%2C+Portugal%2E)
26. Laura Jehl, Patrick Simianer, Julian Hitschler, and Stefan Riezler. 2015. The Heidelberg university English-German translation system for IWSLT 2015. In Proc. of IWSLT, Da Nang, Vietnam. [find](https://scholar.google.com/scholar?q=Laura+Jehl%2C+Patrick+Simianer%2C+Julian+Hitschler%2C+and+Stefan+Riezler%2E+2015%2E+The+Heidelberg+university+English%2DGerman+translation+system+for+IWSLT+2015%2E+In+Proc%2E+of+IWSLT%2C+Da+Nang%2C+Vietnam%2E)
27. Maarit Koponen. 2012. Comparing human perceptions of post-editing effort with post-editing operations. In Proc. of WMT, Montréal, Canada. [find](https://scholar.google.com/scholar?q=Maarit+Koponen%2E+2012%2E+Comparing+human+perceptions+of+post%2Dediting+effort+with+post%2Dediting+operations%2E+In+Proc%2E+of+WMT%2C+Montre%CC%81al%2C+Canada%2E)
28. Arle Lommel, Aljoscha Burchardt, Maja Popović, Kim Harris, Eleftherios Avramidis, and Hans Uszkoreit. 2014. Using a new analytic measure for the annotation and analysis of MT errors on real data. In Proc. of EAMT, Dubrovnik, Croatia. [find](https://scholar.google.com/scholar?q=Arle+Lommel%2C+Aljoscha+Burchardt%2C+Maja+Popovic%CC%81%2C+Kim+Harris%2C+Eleftherios+Avramidis%2C+and+Hans+Uszkoreit%2E+2014%2E+Using+a+new+analytic+measure+for+the+annotation+and+analysis+of+MT+errors+on+real+data%2E+In+)
29. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spoken language domains. In Proc. of IWSLT, Da Nang, Vietnam. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong+and+Christopher+D+Manning%2E+2015%2E+Stanford+neural+machine+translation+systems+for+spoken+language+domains%2E+In+Proc%2E+of+IWSLT%2C+Da+Nang%2C+Vietnam%2E)
30. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP, Lisbon, Portugal. [↗](https://doi.org/10.48550/arxiv.1508.04025)
31. Graham Neubig, Makoto Morishita, and Satoshi Nakamura. 2015. Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015. In Proc. of WAT2015, Kyoto, Japan. [↗](https://doi.org/10.48550/arxiv.1510.05203)
32. Maja Popović and Hermann Ney. 2011. Towards automatic error analysis of machine translation output. Computational Linguistics, 37(4):657–688. [↗](https://doi.org/10.1162/coli_a_00072)
33. Maja Popović, Eleftherios Avramidis, Aljoscha Burchardt, Sabine Hunsicker, Sven Schmeier, Cindy Tscherwinka, David Vilar, and Hans Uszkoreit. 2013. Learning from human judgments of machine translation output. In Proc. of MT Summit, Nice, France. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2C+Eleftherios+Avramidis%2C+Aljoscha+Burchardt%2C+Sabine+Hunsicker%2C+Sven+Schmeier%2C+Cindy+Tscherwinka%2C+David+Vilar%2C+and+Hans+Uszkoreit%2E+2013%2E+Learning+from+human+judgments+of+machine+translatio)
34. Maja Popović. 2011. Hjerson: an open source tool for automatic error classification of machine translation output. The Prague Bulletin of Mathematical Linguistic, (96):59–68. [↗](https://doi.org/10.2478/v10108-011-0011-4)
35. Jean Pouget-Abadie, Dzmitry Bahdanau, Bart van Merrienboer, Kyunghyun Cho, and Yoshua Bengio. 2014. Overcoming the curse of sentence length for neural machine translation using automatic segmentation. In Proc. of SSST-8, Doha, Qatar. [find](https://scholar.google.com/scholar?q=Jean+Pouget%2DAbadie%2C+Dzmitry+Bahdanau%2C+Bart+van+Merrienboer%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Overcoming+the+curse+of+sentence+length+for+neural+machine+translation+using+automatic+segmentation%2E)
36. Nicholas Ruiz and Marcello Federico. 2014. Complexity of spoken versus written language for machine translation. In Proc. of EAMT, Dubrovnik, Croatia. [find](https://scholar.google.com/scholar?q=Nicholas+Ruiz+and+Marcello+Federico%2E+2014%2E+Complexity+of+spoken+versus+written+language+for+machine+translation%2E+In+Proc%2E+of+EAMT%2C+Dubrovnik%2C+Croatia%2E)
37. Rico Sennrich, Martin Volk, and Gerold Schneider. 2013. Exploiting synergies between open resources for German dependency parsing, POS-tagging, and morphological analysis. In Proc. of RANLP, Hissar, Bulgaria. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Martin+Volk%2C+and+Gerold+Schneider%2E+2013%2E+Exploiting+synergies+between+open+resources+for+German+dependency+parsing%2C+POS%2Dtagging%2C+and+morphological+analysis%2E+In+Proc%2E+of+RANLP%2C+Hissar%2C+B)
38. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. of AMTA, Boston, US-MA. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Rich+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proc%2E+of+AMTA%2C+Boston%2C+US%2DMA%2E) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
39. Sara Stymne and Lars Ahrenberg. 2012. On the practice of error analysis for machine translation evaluation. In Proc. of LREC, Istanbul, Turkey. [find](https://scholar.google.com/scholar?q=Sara+Stymne+and+Lars+Ahrenberg%2E+2012%2E+On+the+practice+of+error+analysis+for+machine+translation+evaluation%2E+In+Proc%2E+of+LREC%2C+Istanbul%2C+Turkey%2E)
40. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS, Montréal, Canada. [↗](https://doi.org/10.48550/arxiv.1409.3215)
41. Christoph Tillmann, Stephan Vogel, Hermann Ney, Alexander Zubiaga, and Hassan Sawaf. 1997. Accelerated DP based search for statistical translation. In Proc. of Eurospeech, Rhodes, Greece. David Vilar, Jia Xu, Luis Fernando d’Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proc. of LREC, Genoa, Italy. [find](https://scholar.google.com/scholar?q=Christoph+Tillmann%2C+Stephan+Vogel%2C+Hermann+Ney%2C+Alexander+Zubiaga%2C+and+Hassan+Sawaf%2E+1997%2E+Accelerated+DP+based+search+for+statistical+translation%2E+In+Proc%2E+of+Eurospeech%2C+Rhodes%2C+Greece%2E+David+Vilar%2C) [our card](https://md.research.elamur.ai/free/en/papers/vilar-2006-error-analysis-statistical-machine-translation/)
42. Daniel Zeman, Mark Fishel, Jan Berka, and Ondrej Bojar. 2011. Addicter: what is wrong with my translations? The Prague Bulletin of Mathematical Linguistic, (96):79–88. [↗](https://doi.org/10.2478/v10108-011-0013-2)