---
title: "On The Evaluation of Machine Translation — Edunov 2020"
description: "Back-translation (BT) is a standard way to exploit target-side monolingual data in NMT."
authors: ["Sergey Edunov", "Myle Ott", "Marc'Aurelio Ranzato", "Michael Auli"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.253/"
doi: "10.18653/v1/2020.acl-main.253"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/edunov-2020-evaluation-machine-translation-systems-trained/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/edunov-2020-evaluation-machine-translation-systems-trained/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / On The Evaluation of Machine Translation Systems Trained With Back-Translation

# On The Evaluation of Machine Translation Systems Trained With Back-Translation

**Authors:** [Sergey Edunov](https://md.research.elamur.ai/free/en/authors/sergey-edunov/), [Myle Ott](https://md.research.elamur.ai/free/en/authors/myle-ott/), [Marc'Aurelio Ranzato](https://md.research.elamur.ai/free/en/authors/marc-aurelio-ranzato/), [Michael Auli](https://md.research.elamur.ai/free/en/authors/michael-auli/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.253/) · [PDF](https://aclanthology.org/2020.acl-main.253.pdf) · [DOI 10.18653/v1/2020.acl-main.253](https://doi.org/10.18653/v1/2020.acl-main.253) · [OpenAlex](https://openalex.org/W2967600676)

**Citations:** 81 citations (openalex-snapshot · 2026-10-14) · [58](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Back-translation (BT) is a standard way to exploit target-side monolingual data in NMT. However, BLEU gains from BT appear mostly on test sentences whose source is itself a human translation (translationese). Critics have argued that BT therefore helps only because such inputs resemble synthetic training sources. Edunov, Ott, Ranzato and Auli test this claim with fairseq Transformer Big models trained with and without BT (OP = parallel data only). They cover En-De, De-En, En-Ru and Ru-En, using WMT'18/WMT'19 data and the winning Facebook-FAIR systems. They also commissioned professional sentence-level translations and double translations of newstest subsets. These let them separate input effects from reference effects. Human evaluators prefer BT outputs for both naturally occurring and translationese sources. BLEU misses this in the direct portion because its references are translationese, and BT and OP outputs sit about equally close to them. BT outputs are measurably more fluent. The authors therefore recommend reporting a language-model perplexity next to BLEU. The work matters because it defends BT as a technique and exposes a systematic blind spot in reference-based evaluation.

## Key points
- **Training data:** En-De used 5.18M WMT'18 sentence pairs without ParaCrawl, plus sampled BT of 226M German newscrawl sentences. The WMT'19 directions used parallel data including ParaCrawl: 27.7M pairs for English–German and 26M for English–Russian. Monolingual data was 424M English and 76M Russian newscrawl sentences.
- **Test design:** Test sets were split into direct (X→Y\*) and reverse (X\*→Y) portions. For En↔De, 500 source-original and 500 target-original newstest2014 sentences were sampled. Extra double translations (X\*\*, Y\*\*) let the authors check separately whether translationese is easier as input and as reference.
- **Key finding:** Translationese inputs are easier to translate for both BT and OP models, not only for BT models. Human judgement of BT systems correlates poorly with BLEU.
- **Fluency evidence:**
  - In blind pairwise ratings without the source (100 judgements per direction), native speakers of English, German and Russian rated BT significantly more fluent than OP (p=0.05).
  - Language models trained on system outputs also showed that BT outputs are closer to natural target text.
- **Proposed metric:** Score outputs with a single LM trained on Common Crawl (315M English, 284M German and 120M Russian sentences), kept disjoint from the BT monolingual data. In all four directions this LM gave BT lower perplexity than OP (e.g., 57.6 vs 68.6), despite similar BLEU. The authors suggest using it to break BLEU ties.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

58 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 52 not yet.

- [Translation Artifacts in Cross-lingual Transfer Learning](https://md.research.elamur.ai/free/en/papers/artetxe-2020-translation-artifacts-cross-lingual-transfer/)
- [BLEU might be Guilty but References are not Innocent](https://md.research.elamur.ai/free/en/papers/freitag-2020-bleu-might-be-guilty-but/)
- [Statistical Power and Translationese in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/graham-2020-statistical-power-translationese-machine-translation/)
- [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/)
- [Facebook AI's WMT21 News Translation Task Submission](https://md.research.elamur.ai/free/en/papers/tran-2021-facebook-ai-s-wmt21-news/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{edunov2020evaluation,
  title = {{On The Evaluation of Machine Translation Systems Trained With Back-Translation}},
  author = {Edunov, Sergey and Ott, Myle and Ranzato, Marc'Aurelio and Auli, Michael},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.253},
  url = {https://doi.org/10.18653/v1/2020.acl-main.253}
}
```

- [← previous On the Cross-lingual Transferability of Monolingual Representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
- [next → On the Frailty of Universal POS Tags for Neural UD Parsers](https://md.research.elamur.ai/free/en/papers/anderson-2020-frailty-universal-pos-tags-neural/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

37 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv, abs/1809.10853. [↗](https://arxiv.org/abs/1809.10853)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proc%2E+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Mona Baker. 1993. Corpus linguistics and translation studies: Implications and applications. Text and technology: In honour of John Sinclair, 233:250. [find](https://scholar.google.com/scholar?q=Mona+Baker%2E+1993%2E+Corpus+linguistics+and+translation+studies%3A+Implications+and+applications%2E+Text+and+technology%3A+In+honour+of+John+Sinclair%2C+233%3A250%2E)
4. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measure for Machine Translation. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+Meteor%3A+An+automatic+metric+for+mt+evaluation+with+improved+correlation+with+human+judgments%2E+In+In+Proc%2E+of+ACL+Workshop+on+Intrinsic+and+Extrinsic+Evaluation) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
5. Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Ondrej+Bojar+and+Ales+Tamchyna%2E+2011%2E+Improving+translation+model+by+monolingual+data%2E+In+Proc%2E+of+WMT%2E)
6. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+and+Christof+Monz%2E+2018%2E+Findings+of+the+2018+conference+on+machine+translation+%28WMT18%29%2E+In+) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/)
7. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2019. Findings of the 2019 conference on machine translation (wmt19). In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+and+Christof+Monz%2E+2019%2E+Findings+of+the+2019+conference+on+machine+translation+%28wmt19%29%2E+In+) [our card](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
8. Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Isaac+Caswell%2C+Ciprian+Chelba%2C+and+David+Grangier%2E+2019%2E+Tagged+back%2Dtranslation%2E+In+Proc%2E+of+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/caswell-2019-tagged-back-translation/)
9. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+Proc%2E+of+EMNLP%2E)
10. Markus Freitag, Isaac Caswell, and Scott Roy. 2019. Text repair model for neural machine translation. arXiv, abs/1904.04790. [↗](https://arxiv.org/abs/1904.04790)
11. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+Denis+Yarats%2C+and+Yann+N+Dauphin%2E+2017%2E+Convolutional+Sequence+to+Sequence+Learning%2E+In+Proc%2E+of+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)
12. Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In Proc. of 2nd Workshop on Neural Machine Translation and Generation. [find](https://scholar.google.com/scholar?q=Vu+Cong+Duy+Hoang%2C+Philipp+Koehn%2C+Gholamreza+Haffari%2C+and+Trevor+Cohn%2E+2018%2E+Iterative+back%2Dtranslation+for+neural+machine+translation%2E+In+Proc%2E+of+2nd+Workshop+on+Neural+Machine+Translation+and+Gener) [our card](https://md.research.elamur.ai/free/en/papers/hoang-2018-iterative-back-translation-neural-machine/)
13. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2004%2E+Statistical+significance+tests+for+machine+translation+evaluation%2E+In+Proc%2E+of+EMNLP%2E)
14. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Moshe+Koppel+and+Noam+Ordan%2E+2011%2E+Translationese+and+its+dialects%2E+In+Proc%2E+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/koppel-2011-translationese-its-dialects/)
15. David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact on machine translation. In Proc. of MT Summit. [find](https://scholar.google.com/scholar?q=David+Kurokawa%2C+Cyril+Goutte%2C+and+Pierre+Isabelle%2E+2009%2E+Automatic+detection+of+translated+text+and+its+impact+on+machine+translation%2E+In+Proc%2E+of+MT+Summit%2E)
16. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based and neural unsupervised machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Myle+Ott%2C+Alexis+Conneau%2C+Ludovic+Denoyer%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2018%2E+Phrase%2Dbased+and+neural+unsupervised+machine+translation%2E+In+EMNLP%2E)
17. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2011. Language models for machine translation: Original vs. translated texts. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Gennadi+Lembersky%2C+Noam+Ordan%2C+and+Shuly+Wintner%2E+2011%2E+Language+models+for+machine+translation%3A+Original+vs%2E+translated+texts%2E+In+Proc%2E+of+EMNLP%2E)
18. Marco Lui and Timothy Baldwin. 2012. langid. py: An off-the-shelf language identification tool. In Proc. of ACL: Demonstrations. [find](https://scholar.google.com/scholar?q=Marco+Lui+and+Timothy+Baldwin%2E+2012%2E+langid%2E+py%3A+An+off%2Dthe%2Dshelf+language+identification+tool%2E+In+Proc%2E+of+ACL%3A+Demonstrations%2E) [our card](https://md.research.elamur.ai/free/en/papers/lui-2012-langid-py-off-shelf-language/)
19. Qingsong Ma, Johnny Wei, Ondrej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Johnny+Wei%2C+Ondrej+Bojar%2C+and+Yvette+Graham%2E+2019%2E+Results+of+the+wmt19+metrics+shared+task%3A+Segment%2Dlevel+and+strong+mt+systems+pose+big+challenges%2E+In+Proc%2E+of+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/ma-2019-results-wmt19-metrics-shared-task/)
20. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Nathan+Ng%2C+Kyra+Yee%2C+Alexei+Baevski%2C+Myle+Ott%2C+Michael+Auli%2C+and+Sergey+Edunov%2E+2019%2E+Facebook+fair%E2%80%99s+wmt19+news+translation+task+submission%2E+In+Proc%2E+of+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/ng-2019-facebook-fair-s-wmt19-news/)
21. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL: Demonstrations. [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+Alexei+Baevski%2C+Angela+Fan%2C+Sam+Gross%2C+Nathan+Ng%2C+David+Grangier%2C+and+Michael+Auli%2E+2019%2E+fairseq%3A+A+fast%2C+extensible+toolkit+for+sequence+modeling%2E+In+Proc%2E+of+NAACL%3A+Demonst) [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
22. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+David+Grangier%2C+and+Michael+Auli%2E+2018%2E+Scaling+neural+machine+translation%2E+In+Proc%2E+of+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/)
23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL. Alberto Poncelas, Dimitar Sht. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proc%2E+of+ACL%2E+Alberto+Poncelas%2C+Dimitar+Sht%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
24. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018a. Investigating backtranslation in neural machine translation. arXiv, 1804.06189. Alberto Poncelas, Dimitar Sht. [find](https://scholar.google.com/scholar?q=Shterionov%2C+Andy+Way%2C+Gideon+Maillette+de+Buy+Wenniger%2C+and+Peyman+Passban%2E+2018a%2E+Investigating+backtranslation+in+neural+machine+translation%2E+arXiv%2C+1804%2E06189%2E+Alberto+Poncelas%2C+Dimitar+Sht%2E) [our card](https://md.research.elamur.ai/free/en/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/)
25. Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018b. Investigating backtranslation in neural machine translation. arXiv, 1804.06189. [find](https://scholar.google.com/scholar?q=Shterionov%2C+Andy+Way%2C+Gideon+Maillette+de+Buy+Wenniger%2C+and+Peyman+Passban%2E+2018b%2E+Investigating+backtranslation+in+neural+machine+translation%2E+arXiv%2C+1804%2E06189%2E) [our card](https://md.research.elamur.ai/free/en/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/)
26. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proc%2E+of+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016a%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+Proc%2E+of+ACL%2E)
28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016b%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proc%2E+of+ACL%2E)
29. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. of AMTA. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proc%2E+of+AMTA%2E) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
30. Milos Stanojevic and Khalil Sima’an. 2014. Beer: Better evaluation as ranking. In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Milos+Stanojevic+and+Khalil+Sima%E2%80%99an%2E+2014%2E+Beer%3A+Better+evaluation+as+ranking%2E+In+Proc%2E+of+WMT%2E)
31. Sara Stymne. 2017. The effect of translationese on tuning for statistical machine translation. In Proc. of NoDaLiDa. [find](https://scholar.google.com/scholar?q=Sara+Stymne%2E+2017%2E+The+effect+of+translationese+on+tuning+for+statistical+machine+translation%2E+In+Proc%2E+of+NoDaLiDa%2E)
32. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Proc%2E+of+NIPS%2E)
33. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Proc. of WMT. [find](https://scholar.google.com/scholar?q=Antonio+Toral%2C+Sheila+Castilho%2C+Ke+Hu%2C+and+Andy+Way%2E+2018%2E+Attaining+the+unattainable%3F+reassessing+claims+of+human+parity+in+neural+machine+translation%2E+In+Proc%2E+of+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/)
34. Gideon Toury. 2012. Descriptive translation studies and beyond: Revised edition, volume 100. John Benjamins Publishing. [find](https://scholar.google.com/scholar?q=Gideon+Toury%2E+2012%2E+Descriptive+translation+studies+and+beyond%3A+Revised+edition%2C+volume+100%2E+John+Benjamins+Publishing%2E)
35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proc. of NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+Is+All+You+Need%2E+In+Proc%2E+of+NIPS%2E)
36. Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. arXiv, abs/1906.08069. [↗](https://arxiv.org/abs/1906.08069) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-effect-translationese-machine-translation-test/)
37. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv, abs/1904.09675. [↗](https://arxiv.org/abs/1904.09675)