---
title: "On Evaluation of Adversarial — paper by Michel et al. (2019)"
description: "Using untargeted word-substitution attacks on machine translation (cs-en, de-en, fr-en)… 44.21 chrF. 39 references, cited by 56 papers here."
authors: ["Paul Michel", "Xian Li", "Graham Neubig", "Juan Pino"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1314/"
doi: "10.18653/v1/n19-1314"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/michel-2019-evaluation-adversarial-perturbations-sequence-sequence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/michel-2019-evaluation-adversarial-perturbations-sequence-sequence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models

# On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models

**Authors:** [Paul Michel](https://md.research.elamur.ai/free/en/authors/paul-michel/), [Xian Li](https://md.research.elamur.ai/free/en/authors/xian-li/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) and [Juan Pino](https://md.research.elamur.ai/free/en/authors/juan-pino/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2019](https://md.research.elamur.ai/free/en/venues/naacl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1314/) · [PDF](https://aclanthology.org/N19-1314.pdf) · [DOI 10.18653/v1/n19-1314](https://doi.org/10.18653/v1/n19-1314) · [OpenAlex](https://openalex.org/W2922293812)

**Citations:** 106 citations (OpenAlex, 2026-10-14) · [56](https://md.research.elamur.ai/free/en/papers/michel-2019-evaluation-adversarial-perturbations-sequence-sequence/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Adversarial attacks on seq2seq models are usually judged only by how much they degrade the output. That ignores whether the altered input still means the same thing. If the source meaning changes, a different translation is expected, so this is no model failure. Using untargeted word-substitution attacks on machine translation (cs-en, de-en, fr-en), the authors propose an evaluation framework that scores both how well source meaning is kept and how much target quality drops. They also design a 6-level human similarity scale and test automatic metrics as proxies for it. chrF agrees with human ratings better than BLEU or METEOR. Under this framework, unconstrained gradient-based substitutions often distort the source meaning. Adding constraints (nearest-neighbour substitutions, or CharSwap typo-like character swaps) yields perturbations that keep meaning better but still disrupt translations. Adversarial training with CharSwap examples makes a word-based LSTM more robust without hurting clean test accuracy. Unconstrained adversarial training lacks that benefit. The work matters because it provides a principled way to report attack results.

## Key points
- **Scoring:** The relative target score decrease d_tgt is combined with source similarity s_src into a single success score, S = s_src + d_tgt. An attack with S > 1 damages the target more than it damages the source.
- **Metric choice:** chrF (chrF2) had the best Pearson correlation with human judgments, ahead of BLEU and METEOR. RDchrF is used as the main robustness measure.
- **Constraints:** CharSwap was the only constraint that consistently achieved an average success above 1.
- **Clean test performance (word-based LSTM, α = 1.0):**
  - CharSwap-adv stays close to the baseline, e.g. cs-en 43.74 vs 44.21 chrF.
  - Unconstrained-adv falls to 41.38 chrF.
- **Robustness:** CharSwap-adv lowers RDchrF under CharSwap attacks from about 24 to about 16. The evaluation toolkit is released as teapot-nlp.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations
- [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) — Company, London, United Kingdom · 39 works · 11,541 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Paul Michel, Xian Li, Graham Neubig and Juan Pino. 2019. On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{michel2019evaluation,
  title = {{On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models}},
  author = {Paul Michel, Xian Li, Graham Neubig and Juan Pino},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1314},
  url = {https://doi.org/10.18653/v1/n19-1314}
}
```

- [← previous Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations](https://md.research.elamur.ai/free/en/papers/gu-2019-improved-zero-shot-neural-machine/)
- [next → Context-Aware Monolingual Repair for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/voita-2019-context-aware-monolingual-repair-neural/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

39 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. [Generating natural language adversarial examples](https://md.research.elamur.ai/free/en/papers/alzantot-2018-generating-natural-language-adversarial-examples/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896.
2. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
3. Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316. — [arXiv:1604.07316](https://arxiv.org/abs/1604.07316) [find](https://scholar.google.com/scholar?q=Mariusz+Bojarski%2C+Davide+Del+Testa%2C+Daniel+Dworakowski%2C+Bernhard+Firner%2C+Beat+Flepp%2C+Prasoon+Goyal%2C+Lawrence+D+Jackel%2C+Mathew+Monfort%2C+Urs+Muller%2C+Jiakai+Zhang%2C+et+al%2E+2016%2E+End+to+end+learning+for+se)
4. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. [Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14.
5. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2016. The iwslt 2016 evaluation campaign. [find](https://scholar.google.com/scholar?q=Mauro+Cettolo%2C+Jan+Niehues%2C+Sebastian+Stu%CC%88ker%2C+Luisa+Bentivogli%2C+Roldano+Cattoni%2C+and+Marcello+Federico%2E+2016%2E+The+iwslt+2016+evaluation+campaign%2E)
6. Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin-Yu Chen, and Cho-Jui Hsieh. 2018. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. arXiv preprint arXiv:1803.01128. — [arXiv:1803.01128](https://arxiv.org/abs/1803.01128) [find](https://scholar.google.com/scholar?q=Minhao+Cheng%2C+Jinfeng+Yi%2C+Huan+Zhang%2C+Pin%2DYu+Chen%2C+and+Cho%2DJui+Hsieh%2E+2018%2E+Seq2sick%3A+Evaluating+the+robustness+of+sequence%2Dto%2Dsequence+models+with+adversarial+examples%2E+arXiv+preprint+arXiv%3A1803%2E0112)
7. Moustapha Cissé, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. 2017. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Moustapha+Cisse%CC%81%2C+Piotr+Bojanowski%2C+Edouard+Grave%2C+Yann+Dauphin%2C+and+Nicolas+Usunier%2E+2017%2E+Parseval+networks%3A+Improving+robustness+to+adversarial+examples%2E+In+International+Conference+on+Machine+Lear)
8. Michael Denkowski and Alon Lavie. 2014. [Meteor universal: Language specific translation evaluation for any target language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/). In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
9. Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018a. [On adversarial examples for character-level neural machine translation](https://md.research.elamur.ai/free/en/papers/ebrahimi-2018-adversarial-examples-character-level-neural/). In COLING.
10. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018b. [Hotflip: White-box adversarial examples for text classification](https://md.research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31–36.
11. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. [Explaining and harnessing adversarial examples](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/). Proceedings of the International Conference on Learning Representations (ICLR).
12. Isao Goto, Ka-Po Chow, Bin Lu, Eiichiro Sumita, and Benjamin Ka-Yin T’sou. 2013. Overview of the patent machine translation task at the ntcir-10 workshop. In NII Test Collection for IR Systems. [find](https://scholar.google.com/scholar?q=Isao+Goto%2C+Ka%2DPo+Chow%2C+Bin+Lu%2C+Eiichiro+Sumita%2C+and+Benjamin+Ka%2DYin+T%E2%80%99sou%2E+2013%2E+Overview+of+the+patent+machine+translation+task+at+the+ntcir%2D10+workshop%2E+In+NII+Test+Collection+for+IR+Systems%2E)
13. Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel. 2016. Adversarial perturbations against deep neural networks for malware classification. arXiv preprint arXiv:1606.04435. — [arXiv:1606.04435](https://arxiv.org/abs/1606.04435) [find](https://scholar.google.com/scholar?q=Kathrin+Grosse%2C+Nicolas+Papernot%2C+Praveen+Manoharan%2C+Michael+Backes%2C+and+Patrick+McDaniel%2E+2016%2E+Adversarial+perturbations+against+deep+neural+networks+for+malware+classification%2E+arXiv+preprint+arXiv)
14. Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google’s perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138. — [arXiv:1702.08138](https://arxiv.org/abs/1702.08138) [find](https://scholar.google.com/scholar?q=Hossein+Hosseini%2C+Sreeram+Kannan%2C+Baosen+Zhang%2C+and+Radha+Poovendran%2E+2017%2E+Deceiving+google%E2%80%99s+perspective+api+built+for+detecting+toxic+comments%2E+arXiv+preprint+arXiv%3A1702%2E08138%2E+%E2%80%94+%5BarXiv%3A1702%2E08138%5D)
15. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. [Deep unordered composition rivals syntactic methods for text classification](https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural L…
16. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. [Adversarial example generation with syntactically controlled paraphrase networks](https://md.research.elamur.ai/free/en/papers/iyyer-2018-adversarial-example-generation-syntactically-controlled/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
17. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031.
18. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
19. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondr%CC%8Cej+Bojar%2C+Alexandra+Const)
20. J. Zico Kolter and Eric Wong. 2017. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv preprint arXiv:1711.00851. — [arXiv:1711.00851](https://arxiv.org/abs/1711.00851) [find](https://scholar.google.com/scholar?q=J%2E+Zico+Kolter+and+Eric+Wong%2E+2017%2E+Provable+defenses+against+adversarial+examples+via+the+convex+outer+adversarial+polytope%2E+arXiv+preprint+arXiv%3A1711%2E00851%2E+%E2%80%94+%5BarXiv%3A1711%2E00851%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
21. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421.
22. Xiaoyi Ma and Christopher Cieri. 2006. Corpus support for machine translation at ldc. In Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Xiaoyi+Ma+and+Christopher+Cieri%2E+2006%2E+Corpus+support+for+machine+translation+at+ldc%2E+In+Language+Resources+and+Evaluation+Conference%2E)
23. Leo X McCusker, Philip B Gough, and Randolph G Bias. 1981. Word recognition inside out and outside in. Journal of Experimental Psychology: Human Perception and Performance, 7(3):538. [find](https://scholar.google.com/scholar?q=Leo+X+McCusker%2C+Philip+B+Gough%2C+and+Randolph+G+Bias%2E+1981%2E+Word+recognition+inside+out+and+outside+in%2E+Journal+of+Experimental+Psychology%3A+Human+Perception+and+Performance%2C+7%283%29%3A538%2E)
24. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: A simple and accurate method to fool deep neural networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2574–2582. [find](https://scholar.google.com/scholar?q=Seyed%2DMohsen+Moosavi%2DDezfooli%2C+Alhussein+Fawzi%2C+and+Pascal+Frossard%2E+2016%2E+Deepfool%3A+A+simple+and+accurate+method+to+fool+deep+neural+networks%2E+2016+IEEE+Conference+on+Computer+Vision+and+Pattern+Reco)
25. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353. [find](https://scholar.google.com/scholar?q=Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Norman+Sadeh%2C+Carolyn+Rose%2C+and+Graham+Neubig%2E+2018%2E+Stress+test+evaluation+for+natural+language+inference%2E+In+Proceedings+of+the+27th+International+Conference+o)
26. Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. In Military Communications Conference, MILCOM 2016-2016 IEEE, pages 49–54. IEEE. [find](https://scholar.google.com/scholar?q=Nicolas+Papernot%2C+Patrick+McDaniel%2C+Ananthram+Swami%2C+and+Richard+Harang%2E+2016%2E+Crafting+adversarial+input+sequences+for+recurrent+neural+networks%2E+In+Military+Communications+Conference%2C+MILCOM+2016%2D20)
27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318.
28. Maja Popović. 2015. [chrf: character n-gram f-score for automatic mt evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/). In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395.
29. Maja Popović. 2016. [chrf deconstructed: beta parameters and n-gram weights](https://md.research.elamur.ai/free/en/papers/popovic-2016-chrf-deconstructed-beta-parameters-n/). In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 499–504.
30. Matt Post. 2018. [A call for clarity in reporting bleu scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). arXiv preprint arXiv:1804.08771.
31. Sravana Reddy and Kevin Knight. 2016. Obfuscating gender in social media writing. In Proceedings of the First Workshop on NLP and Computational Social Science. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sravana+Reddy+and+Kevin+Knight%2E+2016%2E+Obfuscating+gender+in+social+media+writing%2E+In+Proceedings+of+the+First+Workshop+on+NLP+and+Computational+Social+Science%2E+Association+for+Computational+Linguistic)
32. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018%2E+Semantically+equivalent+adversarial+rules+for+debugging+nlp+models%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Compu)
33. Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. arXiv preprint arXiv:1707.02812. [find](https://scholar.google.com/scholar?q=Suranjana+Samanta+and+Sameep+Mehta%2E+2017%2E+Towards+crafting+text+adversarial+samples%2E+arXiv+preprint+arXiv%3A1707%2E02812%2E)
34. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725.
35. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+E+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+Machine+Learning+Re)
36. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jonathon+Shlens%2C+and+Zbigniew+Wojna%2E+2016%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+2016+IEEE+Conference+on+Computer+Vision+and+Pa)
37. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2013. [Intriguing properties of neural networks](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/). arXiv preprint arXiv:1312.6199.
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS), pages 5998–6008.
39. Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Zhengli+Zhao%2C+Dheeru+Dua%2C+and+Sameer+Singh%2E+2018%2E+Generating+natural+adversarial+examples%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR%29%2E)

*Updated 24 October 2026*