---
title: "Robust Neural Machine Translation with Doubly — Cheng 2019"
description: "NMT systems can produce much worse output when the input changes only slightly."
authors: ["Yong Cheng", "Lu Jiang", "Wolfgang Macherey"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1425/"
doi: "10.18653/v1/p19-1425"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cheng-2019-robust-neural-machine-translation-doubly/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cheng-2019-robust-neural-machine-translation-doubly/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Robust Neural Machine Translation with Doubly Adversarial Inputs

# Robust Neural Machine Translation with Doubly Adversarial Inputs

**Authors:** [Yong Cheng](https://md.research.elamur.ai/free/en/authors/yong-cheng/), Lu Jiang, [Wolfgang Macherey](https://md.research.elamur.ai/free/en/authors/wolfgang-macherey/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1425/) · [PDF](https://aclanthology.org/P19-1425.pdf) · [DOI 10.18653/v1/p19-1425](https://doi.org/10.18653/v1/p19-1425) · [OpenAlex](https://openalex.org/W2963823140)

**Citations:** 227 citations (openalex-snapshot · 2026-10-14) · [94](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
NMT systems can produce much worse output when the input changes only slightly. In one example, swapping "he" for "she" in a Chinese sentence derailed a Transformer translation. The authors address this weakness with a white-box adversarial training scheme for word-level Transformer models.

Their gradient-guided procedure, AdvGen, replaces a sampled fraction of words with substitutes. It favours words whose embedding shift best aligns with the gradient of the translation loss. Candidate substitutes come from a small set proposed by a bidirectional language model, which keeps the search cheap and the replacements plausible.

The method works in two stages:
- **Attack:** perturbed source sentences attack the encoder.
- **Defence:** adversarially altered decoder inputs, chosen using a mix of language-model and NMT predictions, train the model to tolerate errors caused by the noisy source.

Training uses only parallel data. On Chinese-English and English-German, the approach beats Transformer on clean benchmarks and degrades less on noisy input. This shows that adversarial training can improve both accuracy and robustness.

## Key points
- Reported gains over Transformer are 2.8 BLEU on Chinese-English and 1.6 BLEU on English-German, measured on standard clean test sets.
- In the Chinese-English ablation, the full model (clean loss, robust loss and language-model loss) reaches 46.95 BLEU, compared with 44.59 for the clean-loss-only baseline. Removing the adversarial target inputs causes the largest drop, 1.87 BLEU.
- Replacement ratios have only a mild effect on results. The best validation setting replaces 25% of source words (γ_src = 0.25) and 50% of target inputs (γ_trg = 0.50). Any non-zero γ_trg helped, while larger source ratios hurt.
- Using only parallel corpora, the method outperforms back-translation with extra monolingual data (Sennrich et al., 2016b). Adding monolingual data improves it further.
- The work differs from earlier robustness efforts that relied on black-box or synthetic noise and from the white-box attacks of Ebrahimi et al. (2018a) on character-level models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 91 not yet.

- [Contextualized Perturbation for Textual Adversarial Attack](https://md.research.elamur.ai/free/en/papers/li-2021-contextualized-perturbation-textual-adversarial-attack/)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/raunak-2021-curious-case-hallucinations-neural-machine/)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/zhu-2019-freelb-enhanced-adversarial-training-natural/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cheng2019robust,
  title = {{Robust Neural Machine Translation with Doubly Adversarial Inputs}},
  author = {Cheng, Yong and Jiang, Lu and Macherey, Wolfgang},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1425},
  url = {https://doi.org/10.18653/v1/p19-1425}
}
```

- [← previous Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
- [next → S2ORC: The Semantic Scholar Open Research Corpus](https://md.research.elamur.ai/free/en/papers/lo-2019-s2orc-semantic-scholar-open-research/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Moustafa+Alzantot%2C+Yash+Sharma%2C+Ahmed+Elgohary%2C+Bo%2DJhang+Ho%2C+Mani+Srivastava%2C+and+Kai%2DWei+Chang%2E+2018%2E+Generating+natural+language+adversarial+examples%2E+In+Empirical+Methods+in+Natural+Language+Proces) [our card](https://md.research.elamur.ai/free/en/papers/alzantot-2018-generating-natural-language-adversarial-examples/)
2. Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+KyungHyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. arXiv preprint arXiv:1808.07561. [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Mia+Xu+Chen%2C+Orhan+Firat%2C+Yuan+Cao%2C+and+Yonghui+Wu%2E+2018%2E+Training+deeper+neural+machine+translation+models+with+transparent+attention%2E+arXiv+preprint+arXiv%3A1808%2E07561%2E)
4. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+International+Conference+on+Learning+Representations%2E)
5. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, et al. 2018. The best of both worlds: Combining recent advances in neural machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mia+Xu+Chen%2C+Orhan+Firat%2C+Ankur+Bapna%2C+Melvin+Johnson%2C+Wolfgang+Macherey%2C+George+Foster%2C+Llion+Jones%2C+Niki+Parmar%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+et+al%2E+2018%2E+The+best+of+both+worlds%3A+Combining+recent+a) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/)
6. Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Zhaopeng+Tu%2C+Fandong+Meng%2C+Junjie+Zhai%2C+and+Yang+Liu%2E+2018%2E+Towards+robust+neural+machine+translation%2E+In+Association+for+Computational+Linguistics%2E)
7. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Wei+Xu%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Semi%2Dsupervised+learning+for+neural+machine+translation%2E+In+Association+for+Computational+Linguistics%2E)
8. Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018a. On adversarial examples for character-level neural machine translation. In Proceedings of COLING. [find](https://scholar.google.com/scholar?q=Javid+Ebrahimi%2C+Daniel+Lowd%2C+and+Dejing+Dou%2E+2018a%2E+On+adversarial+examples+for+character%2Dlevel+neural+machine+translation%2E+In+Proceedings+of+COLING%2E)
9. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018b. Hotflip: White-box adversarial examples for text classification. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Javid+Ebrahimi%2C+Anyi+Rao%2C+Daniel+Lowd%2C+and+Dejing+Dou%2E+2018b%2E+Hotflip%3A+White%2Dbox+adversarial+examples+for+text+classification%2E+In+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/)
10. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+Empirical+Methods+in+Natural+Language+Processing%2E)
11. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marzieh+Fadaee%2C+Arianna+Bisazza%2C+and+Christof+Monz%2E+2017%2E+Data+augmentation+for+low%2Dresource+neural+machine+translation%2E+In+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/en/papers/fadaee-2017-data-augmentation-low-resource-neural/)
12. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+Denis+Yarats%2C+and+Yann+N+Dauphin%2E+2017%2E+Convolutional+sequence+to+sequence+learning%2E+In+International+Conference+on+Machine+Learning%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)
13. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jonathon+Shlens%2C+and+Christian+Szegedy%2E+2015%2E+Explaining+and+harnessing+adversarial+examples%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/)
14. Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. 2016. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828. [find](https://scholar.google.com/scholar?q=Di+He%2C+Yingce+Xia%2C+Tao+Qin%2C+Liwei+Wang%2C+Nenghai+Yu%2C+Tie%2DYan+Liu%2C+and+Wei%2DYing+Ma%2E+2016%2E+Dual+learning+for+machine+translation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+820%E2%80%93828%2E)
15. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2017. Sockeye: A toolkit for neural machine translation. arXiv preprint arXiv:1712.05690. — [arXiv:1712.05690](https://arxiv.org/abs/1712.05690) [find](https://scholar.google.com/scholar?q=Felix+Hieber%2C+Tobias+Domhan%2C+Michael+Denkowski%2C+David+Vilar%2C+Artem+Sokolov%2C+Ann+Clifton%2C+and+Matt+Post%2E+2017%2E+Sockeye%3A+A+toolkit+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1712%2E05690%2E+%E2%80%94+%5BarX)
16. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Robin+Jia+and+Percy+Liang%2E+2017%2E+Adversarial+examples+for+evaluating+reading+comprehension+systems%2E+In+Empirical+Methods+in+Natural+Language+Processing%2E) [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
17. Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. 2015. Self-paced curriculum learning. In AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Lu+Jiang%2C+Deyu+Meng%2C+Qian+Zhao%2C+Shiguang+Shan%2C+and+Alexander+G+Hauptmann%2E+2015%2E+Self%2Dpaced+curriculum+learning%2E+In+AAAI+Conference+on+Artificial+Intelligence%2E)
18. Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Lu+Jiang%2C+Zhengyuan+Zhou%2C+Thomas+Leung%2C+Li%2DJia+Li%2C+and+Li+Fei%2DFei%2E+2018%2E+Mentornet%3A+Learning+data%2Ddriven+curriculum+for+very+deep+neural+networks+on+corrupted+labels%2E+In+International+Conference+on+Ma)
19. Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise in machine translation. arXiv preprint arXiv:1902.01509. [find](https://scholar.google.com/scholar?q=Vladimir+Karpukhin%2C+Omer+Levy%2C+Jacob+Eisenstein%2C+and+Marjan+Ghazvininejad%2E+2019%2E+Training+on+synthetic+noise+improves+robustness+to+natural+noise+in+machine+translation%2E+arXiv+preprint+arXiv%3A1902%2E0150)
20. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+Tianlin+Shi%2C+Se%CC%81bastien+Jean%2C+Alan+Ritter%2C+and+Dan+Jurafsky%2E+2017%2E+Adversarial+learning+for+neural+dialogue+generation%2E+In+Empirical+Methods+in+Natural+Language+Processing%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/)
21. Hairong Liu, Mingbo Ma, Liang Huang, Hao Xiong, and Zhongjun He. 2018. Robust neural machine translation with joint textual and phonetic embedding. arXiv preprint arXiv:1810.06729. [find](https://scholar.google.com/scholar?q=Hairong+Liu%2C+Mingbo+Ma%2C+Liang+Huang%2C+Hao+Xiong%2C+and+Zhongjun+He%2E+2018%2E+Robust+neural+machine+translation+with+joint+textual+and+phonetic+embedding%2E+arXiv+preprint+arXiv%3A1810%2E06729%2E)
22. Paul Michel and Graham Neubig. 2018. Mtnt: A testbed for machine translation of noisy text. arXiv preprint arXiv:1809.00388. [find](https://scholar.google.com/scholar?q=Paul+Michel+and+Graham+Neubig%2E+2018%2E+Mtnt%3A+A+testbed+for+machine+translation+of+noisy+text%2E+arXiv+preprint+arXiv%3A1809%2E00388%2E) [our card](https://md.research.elamur.ai/free/en/papers/michel-2018-mtnt-testbed-machine-translation-noisy/)
23. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Takeru+Miyato%2C+Andrew+M+Dai%2C+and+Ian+Goodfellow%2E+2017%2E+Adversarial+training+methods+for+semi%2Dsupervised+text+classification%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/miyato-2016-adversarial-training-methods-semi-supervised/)
24. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Seyed%2DMohsen+Moosavi%2DDezfooli%2C+Alhussein+Fawzi%2C+and+Pascal+Frossard%2E+2016%2E+Deepfool%3A+a+simple+and+accurate+method+to+fool+deep+neural+networks%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recogn)
25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Neural machine translation of rare words with subword units. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016c%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Association+for+Computational+Linguistics%2E)
26. Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In International Workshop on Spoken Language Translation. [find](https://scholar.google.com/scholar?q=Matthias+Sperber%2C+Jan+Niehues%2C+and+Alex+Waibel%2E+2017%2E+Toward+robust+neural+machine+translation+for+noisy+input+sequences%2E+In+International+Workshop+on+Spoken+Language+Translation%2E)
27. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
28. Christian Szegedy, Wojciech Zaremba, Sutskever Ilya, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wojciech+Zaremba%2C+Sutskever+Ilya%2C+Joan+Bruna%2C+Dumitru+Erhan%2C+Ian+Goodfellow%2C+and+Rob+Fergus%2E+2014%2E+Intriguing+properties+of+neural+networks%2E+In+International+Conference+on+Machine+L) [our card](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/)
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
30. Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. Switchout: an efficient data augmentation algorithm for neural machine translation. In Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Hieu+Pham%2C+Zihang+Dai%2C+and+Graham+Neubig%2E+2018%2E+Switchout%3A+an+efficient+data+augmentation+algorithm+for+neural+machine+translation%2E+In+Empirical+Methods+in+Natural+Language+Processing%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-switchout-efficient-data-augmentation-algorithm/)
31. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a methof for automatic evaluation of machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+methof+for+automatic+evaluation+of+machine+translation%2E+In+Association+for+Computational+Linguistics%2E)
32. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+the+gap+between+human+and+machi)
33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016a%2E+Edinburgh+neural+machine+translation+systems+for+wmt+16%2E+arXiv+preprint+arXiv%3A1606%2E02891%2E) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/)
34. Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhengli+Zhao%2C+Dheeru+Dua%2C+and+Sameer+Singh%2E+2018%2E+Generating+natural+adversarial+examples%2E+In+International+Conference+on+Learning+Representations%2E)
35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving nerual machine translation models with monolingual data. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016b%2E+Improving+nerual+machine+translation+models+with+monolingual+data%2E+In+Association+for+Computational+Linguistics%2E)