---
title: "Learning to Deceive with Attention-Based Explanations (2020)"
description: "This paper tests whether such explanations can be trusted. 96.4% accuracy. 23 references, cited by 70 papers here."
authors: ["Danish Pruthi", "Mansi Gupta", "Bhuwan Dhingra", "Graham Neubig", "Zachary C. Lipton"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.432/"
doi: "10.18653/v1/2020.acl-main.432"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pruthi-2020-learning-deceive-attention-based-explanations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pruthi-2020-learning-deceive-attention-based-explanations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Learning to Deceive with Attention-Based Explanations

# Learning to Deceive with Attention-Based Explanations

**Authors:** [Danish Pruthi](https://md.research.elamur.ai/free/en/authors/danish-pruthi/), Mansi Gupta, [Bhuwan Dhingra](https://md.research.elamur.ai/free/en/authors/bhuwan-dhingra/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) and [Zachary C. Lipton](https://md.research.elamur.ai/free/en/authors/zachary-c-lipton/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.432/) · [PDF](https://aclanthology.org/2020.acl-main.432.pdf) · [DOI 10.18653/v1/2020.acl-main.432](https://doi.org/10.18653/v1/2020.acl-main.432) · [OpenAlex](https://openalex.org/W2973291083)

**Citations:** 157 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/pruthi-2020-learning-deceive-attention-based-explanations/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Attention weights are often presented as explanations of what drives a neural model's output, including in fairness audits such as checking whether occupation classifiers depend on gender. This paper tests whether such explanations can be trusted. The authors add a penalty to the training loss that punishes attention placed on a chosen set of "impermissible" tokens, such as gender words. For multi-head models like BERT, the penalty applies to either the mean or the maximum attention across heads. They evaluate the approach on four binary classification datasets and four sequence-to-sequence tasks. Attention on the forbidden tokens drops sharply, yet models keep using those tokens and lose surprisingly little accuracy. In a human study, NLP-literate participants shown the manipulated attention concluded that a gender-biased model ignored gender. They also rated these explanations as nearly as believable as the original ones. Analysis shows how the models get around the penalty: recurrent layers move the information to neighboring positions, and simpler models enlarge the embeddings of the forbidden tokens. The findings warn that attention is an unreliable auditing tool.

## Key points
- **Penalty and datasets:** The added term is R = −λ·log(1 − αᵀm), where m marks impermissible tokens. Classification datasets are:
  - CommonCrawl Biographies (physician vs surgeon), with gender indicators at 6.5% of tokens
  - Wikipedia Biographies (gender identification), at 7.6%
  - SST + Wikipedia (sentiment), where the SST sentence is 45.5% of tokens
  - Reference Letters (acceptance prediction), where Percentile/Rank make up 1.6%
- **Occupation task setup:** Over 80% of surgeons in the data are male, and the minority classes were downsampled tenfold. Models reach 96.4% accuracy, which falls to 93.8% when gender pronouns are anonymized, so gender carries real predictive signal.
- **Human study:** With original attention, 66% of participants judged the prediction gender-influenced, versus 0% with the manipulated attention. Trust scores out of 4 were 3.00 for original attention, 2.67 for the manipulated version, and 1.00 for the KL-based adversarial attention of Wiegreffe and Pinter (2019). Annotator agreement was very high (Fleiss' Kappa 0.97).
- **Workarounds:** With impermissible-token attention zeroed after training, the BiLSTM gender classifier still gets over 99% of instances right. The Embedding model drops to about 50%. In seq2seq tasks, attention maps shift by one or two positions, and the mechanism used varies with the random seed.
- **Resources:** Code and datasets are released on GitHub at danishpruthi/deceptive-attention.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 34 works · 51,167 citations
- [Birla Institute of Technology and Science, Pilani](https://md.research.elamur.ai/free/en/universities/birla-institute-of-technology-and-science-pilani/) — University, Pilāni, India · 8 works · 40,950 citations

Also on the byline: [Twitter](https://ror.org/04wt43v05).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works

## Cite

Cite as: Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig and Zachary C. Lipton. 2020. Learning to Deceive with Attention-Based Explanations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pruthi2020learning,
  title = {{Learning to Deceive with Attention-Based Explanations}},
  author = {Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig and Zachary C. Lipton},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.432},
  url = {https://doi.org/10.18653/v1/2020.acl-main.432}
}
```

- [← previous Understanding the Difficulty of Training Transformers](https://md.research.elamur.ai/free/en/papers/liu-2020-understanding-difficulty-training-transformers/)
- [next → Neural CRF Model for Sentence Alignment in Text Simplification](https://md.research.elamur.ai/free/en/papers/jiang-2020-neural-crf-model-sentence-alignment/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473.
2. Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence classification with human attention. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 302–312. [find](https://scholar.google.com/scholar?q=Maria+Barrett%2C+Joachim+Bingel%2C+Nora+Hollenstein%2C+Marek+Rei%2C+and+Anders+S%C3%B8gaard%2E+2018%2E+Sequence+classification+with+human+attention%2E+In+Proceedings+of+the+22nd+Conference+on+Computational+Natural+Langu)
3. Gino Brunner, Yang Liu, Damián Pascual, Oliver Richter, and Roger Wattenhofer. 2019. On the validity of self-attention as explanation in transformer models. arXiv preprint arXiv:1908.04211. — [arXiv:1908.04211](https://arxiv.org/abs/1908.04211) [find](https://scholar.google.com/scholar?q=Gino+Brunner%2C+Yang+Liu%2C+Damia%CC%81n+Pascual%2C+Oliver+Richter%2C+and+Roger+Wattenhofer%2E+2019%2E+On+the+validity+of+self%2Dattention+as+explanation+in+transformer+models%2E+arXiv+preprint+arXiv%3A1908%2E04211%2E+%E2%80%94+%5BarXiv%3A)
4. Wenhu Chen, Evgeny Matusov, Shahram Khadivi, and Jan-Thorsten Peter. 2016. Guided alignment training for topic-aware neural machine translation. arXiv preprint arXiv:1607.01628. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Evgeny+Matusov%2C+Shahram+Khadivi%2C+and+Jan%2DThorsten+Peter%2E+2016%2E+Guided+alignment+training+for+topic%2Daware+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1607%2E01628%2E)
5. Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pages 3504–3512. [find](https://scholar.google.com/scholar?q=Edward+Choi%2C+Mohammad+Taha+Bahadori%2C+Jimeng+Sun%2C+Joshua+Kulas%2C+Andy+Schuetz%2C+and+Walter+Stewart%2E+2016%2E+Retain%3A+An+interpretable+predictive+model+for+healthcare+using+reverse+time+attention+mechanism%2E+)
6. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. arXiv preprint arXiv:1901.09451. — [arXiv:1901.09451](https://arxiv.org/abs/1901.09451) [find](https://scholar.google.com/scholar?q=Maria+De%2DArteaga%2C+Alexey+Romanov%2C+Hanna+Wallach%2C+Jennifer+Chayes%2C+Christian+Borgs%2C+Alexandra+Chouldechova%2C+Sahin+Geyik%2C+Krishnaram+Kenthapadi%2C+and+Adam+Tauman+Kalai%2E+2019%2E+Bias+in+bios%3A+A+case+study+o)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). North American Chapter of the Association for Computational Linguistics (NAACL).
8. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A+Smith%2E+2013%2E+A+simple%2C+fast%2C+and+effective+reparameterization+of+ibm+model+2%2E+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL%29%2E)
9. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. [Multi30k: Multilingual english-german image descriptions](https://md.research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/). arXiv preprint arXiv:1605.00459.
10. Andrea Galassi, Marco Lippi, and Paolo Torroni. 2019. Attention, please! a critical review of neural attention models in natural language processing. arXiv preprint arXiv:1902.02181. [find](https://scholar.google.com/scholar?q=Andrea+Galassi%2C+Marco+Lippi%2C+and+Paolo+Torroni%2E+2019%2E+Attention%2C+please%21+a+critical+review+of+neural+attention+models+in+natural+language+processing%2E+arXiv+preprint+arXiv%3A1902%2E02181%2E)
11. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5-6):602–610. [find](https://scholar.google.com/scholar?q=Alex+Graves+and+Ju%CC%88rgen+Schmidhuber%2E+2005%2E+Frame%2Dwise+phoneme+classification+with+bidirectional+lstm+and+other+neural+network+architectures%2E+Neural+Networks%2C+18%285%2D6%29%3A602%E2%80%93610%2E)
12. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to transduce with unbounded memory. In Advances in neural information processing systems, pages 1828– 1836. [find](https://scholar.google.com/scholar?q=Edward+Grefenstette%2C+Karl+Moritz+Hermann%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Learning+to+transduce+with+unbounded+memory%2E+In+Advances+in+neural+information+processing+systems%2C+pages+1828%E2%80%93+1836%2E)
13. Sarthak Jain, Ramin Mohammadi, and Byron C Wallace. 2019. [Attention is not explanation](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/). North American Chapter of the Association for Computational Linguistics (NAACL).
14. Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 29– 38. ACM. [find](https://scholar.google.com/scholar?q=Vivian+Lai+and+Chenhao+Tan%2E+2019%2E+On+human+predictions+with+explanations+and+predictions+of+machine+learning+models%3A+A+case+study+on+deception+detection%2E+In+Proceedings+of+the+Conference+on+Fairness%2C+)
15. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2016%2E+Understanding+neural+networks+through+representation+erasure%2E+arXiv+preprint+arXiv%3A1612%2E08220%2E)
16. Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. [Neural machine translation with supervised attention](https://md.research.elamur.ai/free/en/papers/liu-2016-neural-machine-translation-supervised-attention/). arXiv preprint arXiv:1609.04186.
17. Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614–1623. [find](https://scholar.google.com/scholar?q=Andre+Martins+and+Ramon+Astudillo%2E+2016%2E+From+softmax+to+sparsemax%3A+A+sparse+model+of+attention+and+multi%2Dlabel+classification%2E+In+International+Conference+on+Machine+Learning%2C+pages+1614%E2%80%931623%2E)
18. Sofia Serrano and Noah A Smith. 2019. [Is attention interpretable?](https://md.research.elamur.ai/free/en/papers/serrano-2019-is-attention-interpretable/) 57th annual meeting of the Association for Computational Linguistics (ACL).
19. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008.
21. Sarah Wiegreffe and Yuval Pinter. 2019. [Attention is not not explanation](https://md.research.elamur.ai/free/en/papers/wiegreffe-2019-attention-is-not-not-explanation/). Proceedings of the 2019 conference on Empirical Methods in Natural Language Processing, EMNLP.
22. Qizhe Xie, Xuezhe Ma, Zihang Dai, and Eduard Hovy. 2017. An interpretable knowledge transfer model for knowledge base completion. arXiv preprint arXiv:1704.05908. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Xuezhe+Ma%2C+Zihang+Dai%2C+and+Eduard+Hovy%2E+2017%2E+An+interpretable+knowledge+transfer+model+for+knowledge+base+completion%2E+arXiv+preprint+arXiv%3A1704%2E05908%2E)
23. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. [Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). arXiv preprint arXiv:1502.03044.

*Updated 24 October 2026*