---
title: "AllenNLP Interpret — paper by Wallace et al. (2019)"
description: "The authors present AllenNLP Interpret, an open-source extension of AllenNLP with three parts… 22 references, cited by 65 papers here."
authors: ["Eric Wallace", "Jens Tuyls", "Junlin Wang", "Sanjay Subramanian", "Matt Gardner", "Sameer Singh"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations"
source: "https://aclanthology.org/D19-3002/"
doi: "10.18653/v1/d19-3002"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wallace-2019-allennlp-interpret-framework-explaining-predictions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wallace-2019-allennlp-interpret-framework-explaining-predictions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

# AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

**Authors:** [Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/), Jens Tuyls, Junlin Wang, [Sanjay Subramanian](https://md.research.elamur.ai/free/en/authors/sanjay-subramanian/), [Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/) and [Sameer Singh](https://md.research.elamur.ai/free/en/authors/sameer-singh/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-3002/) · [PDF](https://aclanthology.org/D19-3002.pdf) · [DOI 10.18653/v1/d19-3002](https://doi.org/10.18653/v1/d19-3002) · [OpenAlex](https://openalex.org/W2970863760)

**Citations:** 115 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/wallace-2019-allennlp-interpret-framework-explaining-predictions/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Neural NLP models are accurate but opaque. Most available explanation code is built for computer vision, tied to a single model or task, or covers only a few techniques, so practitioners rarely use it and researchers struggle to test their methods across systems. The authors present AllenNLP Interpret, an open-source extension of AllenNLP with three parts: ready-made instance-level explanation methods, model- and task-agnostic primitives (chiefly input gradients) for building new methods, and reusable web components for interactive display.

The toolkit focuses on gradient-based techniques, namely three saliency-map variants and two kinds of input-perturbation attacks, because these apply to any differentiable model. The only model-specific code is a short function that turns a model's own prediction into labeled instances. Gradients are then captured through a hook on the embedding layer.

Live demos run five methods on six tasks, including BERT masked language modeling and BiDAF reading comprehension. They expose biases, decision rules and errors. The work makes explaining arbitrary NLP models much cheaper.

## Key points
- **Saliency methods.** The toolkit implements Vanilla Gradient, Integrated Gradients (with an all-zero embedding baseline) and SmoothGrad (Gaussian noise added to embeddings). Each token's gradient is collapsed to one score using the L2 norm.
- **Attack methods.**
  - HotFlip, in its original untargeted form and in a new targeted variant that pushes the model toward a chosen output.
  - Input Reduction, which repeatedly deletes the lowest-gradient word while keeping the prediction unchanged.
- **Supported tasks and models.**
  - Reading comprehension on SQuAD and DROP, using BiDAF and NAQANet.
  - Masked LM with PyTorch Transformers models such as BERT and RoBERTa.
  - Text classification and textual entailment with BiLSTM and self-attention classifiers.
  - NER and coreference resolution.
- **Example findings from the demos.**
  - A SQuAD model relies on question–passage word overlap.
  - An entailment model predicts contradiction when it sees "politics".
  - An NER model links "in downtown" to the Location tag.
  - A sentiment model is misled by "tony hawk style".
- **Extension effort.**
  - Adding SmoothGrad took about ten lines on top of vanilla gradient, about five lines of demo back-end code, and a single-line front-end call.
  - The labeled-instance function for a classifier needs about three lines.

## Interesting ideas
- **Explain predictions rather than gold labels.** The model's own output is used as the target when computing the loss. This lets the same machinery explain any prediction, including on unlabeled data.
- **Split structured outputs into separate instances.** Each NER tag or coreference cluster becomes its own instance, so every part of the output gets its own explanation. This is useful for any sequence-labeling or parsing system.
- **Embedding-agnostic gradients via a hook.** A backward hook on the shared token-embedding interface captures gradients regardless of whether the model uses word vectors, ELMo or BERT. This is a cheap way to add introspection to an existing framework without touching individual models.
- **A substitute vocabulary matrix for contextual embeddings.** Vocabulary items are passed through the model's last context-independent layer (for example, ELMo's Char-CNN), and the results serve as a searchable matrix for word-swap attacks. Gradients still flow through the contextual layers, so this transfers to other discrete-search methods on contextual models.
- **Batch mode alongside the visual demo.** The tool can also run offline over many examples, which allows aggregate statistics such as typical reduced-input length rather than one-off anecdotes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations
- [University of California, Irvine](https://md.research.elamur.ai/free/en/universities/university-of-california-irvine/) — University, Irvine, United States · 29 works · 36,049 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner and Sameer Singh. 2019. AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wallace2019allennlp,
  title = {{AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models}},
  author = {Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner and Sameer Singh},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations},
  year = {2019},
  doi = {10.18653/v1/d19-3002},
  url = {https://doi.org/10.18653/v1/d19-3002}
}
```

- [← previous Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering](https://md.research.elamur.ai/free/en/papers/wallace-2019-trick-me-if-you-can/)
- [next → Data-to-text Generation with Entity Modeling](https://md.research.elamur.ai/free/en/papers/puduppully-2019-data-text-generation-entity-modeling/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In ICLR.
2. Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, et al. 2016. Technical report on the CleverHans v2.1.0 adversarial examples library. arXiv:1610.00768. — [arXiv:1610.00768](https://arxiv.org/abs/1610.00768) [find](https://scholar.google.com/scholar?q=Nicolas+Papernot%2C+Fartash+Faghri%2C+Nicholas+Carlini%2C+Ian+Goodfellow%2C+Reuben+Feinman%2C+Alexey+Kurakin%2C+Cihang+Xie%2C+Yash+Sharma%2C+Tom+Brown%2C+Aurko+Roy%2C+et+al%2E+2016%2E+Technical+report+on+the+CleverHans+v2%2E1%2E)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL.
4. Finale Doshi-Velez and Been Kim. 2017. [Towards a rigorous science of interpretable machine learning](https://md.research.elamur.ai/free/en/papers/doshi-velez-2017-towards-rigorous-science-interpretable-machine/). arXiv preprint arXiv:1702.08608.
5. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In NAACL.
6. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. [DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). In NAACL.
7. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In EMNLP.
8. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. [Why should I trust you?: Explaining the predictions of any classifier](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/). In KDD.
9. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. [HotFlip: White-box adversarial examples for text classification](https://md.research.elamur.ai/free/en/papers/ebrahimi-2018-hotflip-white-box-adversarial-examples/). In ACL.
10. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). In ICLR.
11. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In EMNLP. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Alvin+Grissom+II%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Pathologies+of+neural+models+make+interpretations+difficult%2E+In+EMNLP%2E)
12. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. [Deep inside convolutional networks: Visualising image classification models and saliency maps](https://md.research.elamur.ai/free/en/papers/simonyan-2013-deep-inside-convolutional-networks-visualising/). In ICLR.
13. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. [AllenNLP: A deep semantic natural language processing platform](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/). In ACL Workshop for NLP Open Source Software.
14. Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. 2017. SmoothGrad: removing noise by adding noise. In ICML Workshop on Visualization for Deep Learning. [find](https://scholar.google.com/scholar?q=Daniel+Smilkov%2C+Nikhil+Thorat%2C+Been+Kim%2C+Fernanda+B%2E+Vie%CC%81gas%2C+and+Martin+Wattenberg%2E+2017%2E+SmoothGrad%3A+removing+noise+by+adding+noise%2E+In+ICML+Workshop+on+Visualization+for+Deep+Learning%2E)
15. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+R%2E+Bowman%2C+and+Noah+A%2E+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+NAACL%2E)
16. Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M Rush. 2019. Seq2Seq-Vis: A visual debugging tool for sequence-to-sequence models. IEEE TVCG. [find](https://scholar.google.com/scholar?q=Hendrik+Strobelt%2C+Sebastian+Gehrmann%2C+Michael+Behrisch%2C+Adam+Perer%2C+Hanspeter+Pfister%2C+and+Alexander+M+Rush%2E+2019%2E+Seq2Seq%2DVis%3A+A+visual+debugging+tool+for+sequence%2Dto%2Dsequence+models%2E+IEEE+TVCG%2E)
17. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2016. [Visualizing and understanding recurrent networks](https://md.research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/). In ICLR Workshop Track.
18. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In ICML. [find](https://scholar.google.com/scholar?q=Mukund+Sundararajan%2C+Ankur+Taly%2C+and+Qiqi+Yan%2E+2017%2E+Axiomatic+attribution+for+deep+networks%2E+In+ICML%2E)
19. Gyeongbok Lee, Sungdong Kim, and Seung-won Hwang. 2019. QADiver: Interactive framework for diagnosing QA models. In AAAI Demonstrations. [find](https://scholar.google.com/scholar?q=Gyeongbok+Lee%2C+Sungdong+Kim%2C+and+Seung%2Dwon+Hwang%2E+2019%2E+QADiver%3A+Interactive+framework+for+diagnosing+QA+models%2E+In+AAAI+Demonstrations%2E)
20. Jesse Vig. 2019. Visualizing attention in transformer-based language models. arXiv:1904.02679. — [arXiv:1904.02679](https://arxiv.org/abs/1904.02679) [find](https://scholar.google.com/scholar?q=Jesse+Vig%2E+2019%2E+Visualizing+attention+in+transformer%2Dbased+language+models%2E+arXiv%3A1904%2E02679%2E+%E2%80%94+%5BarXiv%3A1904%2E02679%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1904%2E02679%29)
21. Shusen Liu, Tao Li, Zhimin Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018. Visual interrogation of attention-based models for natural language inference and machine comprehension. In EMNLP. [find](https://scholar.google.com/scholar?q=Shusen+Liu%2C+Tao+Li%2C+Zhimin+Li%2C+Vivek+Srikumar%2C+Valerio+Pascucci%2C+and+Peer%2DTimo+Bremer%2E+2018%2E+Visual+interrogation+of+attention%2Dbased+models+for+natural+language+inference+and+machine+comprehension%2E+In)
22. Eric Wallace, Shi Feng, and Jordan Boyd-Graber. 2018. Interpreting neural networks with nearest neighbors. In EMNLP 2018 Blackbox NLP Workshop. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Shi+Feng%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Interpreting+neural+networks+with+nearest+neighbors%2E+In+EMNLP+2018+Blackbox+NLP+Workshop%2E)

*Updated 24 October 2026*