---
title: "Did the Model Understand the Question? — Mudrakarta 2018"
description: "The authors use these weak spots to build targeted adversarial questions. 32 references, cited by 60 papers here."
authors: ["Pramod Kaushik Mudrakarta", "Ankur Taly", "Mukund Sundararajan", "Kedar Dhamdhere"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P18-1176/"
doi: "10.18653/v1/p18-1176"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mudrakarta-2018-did-model-understand-question/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mudrakarta-2018-did-model-understand-question/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Did the Model Understand the Question?

# Did the Model Understand the Question?

**Authors:** Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan and Kedar Dhamdhere · **Year:** 2018 · **Venue:** [Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2018](https://md.research.elamur.ai/free/en/venues/acl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-1176/) · [PDF](https://aclanthology.org/P18-1176.pdf) · [DOI 10.18653/v1/p18-1176](https://doi.org/10.18653/v1/p18-1176) · [OpenAlex](https://openalex.org/W2798312813)

**Citations:** 151 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/mudrakarta-2018-did-model-understand-question/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether high-accuracy question answering networks actually use the meaningful words in a question. The authors apply Integrated Gradients (IG) to pre-trained models for three tasks, using an empty question as the attribution baseline. The models are a visual QA network (Kazemi and Elqursh, 2017), Neural Programmer for questions over tables, and the reading comprehension model of Yu et al. (2018). The attributions show that these systems often give little weight to key terms such as nouns and adjectives. Instead, they lean on generic or content-free words, or on the context itself.

The authors use these weak spots to build targeted adversarial questions. They also propose an overstability test that tracks how accuracy changes as words are removed from questions. Their attacks sharply degrade the visual and table models, and attribution insights make the ADDSENT attacks of Jia and Liang (2017) more effective.

The work matters because test-set accuracy alone can hide models that are right for the wrong reasons. Attributions help expose this and can point toward better test data.

## Key points
- Adding meaningless prefixes to questions cuts accuracy sharply:
  - the visual QA model falls from 61.1% to 19%;
  - Neural Programmer falls from 33.5% to 3.3%.
- The visual QA model keeps over half of its accuracy even when every question word except "color" is removed.
- Neural Programmer's operation choices are driven more by function words than by content words.
  - Dropping the function words only lowers its accuracy to 28.5%.
  - On the perturbed tables of Pasupat and Liang (2016), its accuracy falls to 23%.
- The authors analyzed 1000 ADDSENT cases, of which 508 fooled the model.
  - Attacks were more likely to fail when the added sentence omitted a highly attributed question noun or adjective: 63% failed in that group versus 40% otherwise.
- Code for generating attributions and reproducing the results is released on GitHub (pramodkaushik/acl18_results).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [University of Chicago](https://md.research.elamur.ai/free/en/universities/university-of-chicago/) — University, Chicago, United States · 28 works · 49,773 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan and Kedar Dhamdhere. 2018. Did the Model Understand the Question?. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mudrakarta2018did,
  title = {{Did the Model Understand the Question?}},
  author = {Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan and Kedar Dhamdhere},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1176},
  url = {https://doi.org/10.18653/v1/p18-1176}
}
```

- [← previous Syntax for Semantic Role Labeling, To Be, Or Not To Be](https://md.research.elamur.ai/free/en/papers/he-2018-syntax-semantic-role-labeling-be/)
- [next → Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer](https://md.research.elamur.ai/free/en/papers/santos-2018-fighting-offensive-language-social-media/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. [Analyzing the behavior of visual question answering models](https://md.research.elamur.ai/free/en/papers/agrawal-2016-analyzing-behavior-visual-question-answering/). arXiv preprint arXiv:1606.07356.
2. Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C Lawrence Zitnick, Dhruv Batra, and Devi Parikh. 2015. Vqa: Visual question answering. arXiv preprint arXiv:1505.00468. [find](https://scholar.google.com/scholar?q=Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Stanislaw+Antol%2C+Margaret+Mitchell%2C+C+Lawrence+Zitnick%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+2015%2E+Vqa%3A+Visual+question+answering%2E+arXiv+preprint+arXiv%3A1505%2E00468%2E)
3. David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. 2010. How to explain individual classification decisions. Journal of Machine Learning Research, pages 1803–1831. [find](https://scholar.google.com/scholar?q=David+Baehrens%2C+Timon+Schroeter%2C+Stefan+Harmeling%2C+Motoaki+Kawanabe%2C+Katja+Hansen%2C+and+Klaus%2DRobert+Mu%CC%88ller%2E+2010%2E+How+to+explain+individual+classification+decisions%2E+Journal+of+Machine+Learning+Resea)
4. Hedi Ben-younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. 2017. Mutan: Multimodal tucker fusion for visual question answering. arXiv preprint arXiv:1705.06676. — [arXiv:1705.06676](https://arxiv.org/abs/1705.06676) [find](https://scholar.google.com/scholar?q=Hedi+Ben%2Dyounes%2C+Re%CC%81mi+Cadene%2C+Matthieu+Cord%2C+and+Nicolas+Thome%2E+2017%2E+Mutan%3A+Multimodal+tucker+fusion+for+visual+question+answering%2E+arXiv+preprint+arXiv%3A1705%2E06676%2E+%E2%80%94+%5BarXiv%3A1705%2E06676%5D%28https%3A%2F%2Farxi)
5. Alexander Binder, Grégoire Montavon, Sebastian Bach, Klaus-Robert Müller, and Wojciech Samek. 2016. Layer-wise relevance propagation for neural networks with local renormalization layers. CoRR. [find](https://scholar.google.com/scholar?q=Alexander+Binder%2C+Gre%CC%81goire+Montavon%2C+Sebastian+Bach%2C+Klaus%2DRobert+Mu%CC%88ller%2C+and+Wojciech+Samek%2E+2016%2E+Layer%2Dwise+relevance+propagation+for+neural+networks+with+local+renormalization+layers%2E+CoRR%2E)
6. Ruth C Fong and Andrea Vedaldi. 2017. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3429–3437. [find](https://scholar.google.com/scholar?q=Ruth+C+Fong+and+Andrea+Vedaldi%2E+2017%2E+Interpretable+explanations+of+black+boxes+by+meaningful+perturbation%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+3429)
7. Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. [Multimodal compact bilinear pooling for visual question answering and visual grounding](https://md.research.elamur.ai/free/en/papers/fukui-2016-multimodal-compact-bilinear-pooling-visual/). arXiv preprint arXiv:1606.01847.
8. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. [Explaining and harnessing adversarial examples](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/). In International Conference on Learning Representations.
9. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). arXiv preprint arXiv:1612.00837.
10. Jia-Hong Huang, Cuong Duc Dao, Modar Alfadly, and Bernard Ghanem. 2017. A novel framework for robustness analysis of visual qa models. arXiv preprint arXiv:1711.06232. — [arXiv:1711.06232](https://arxiv.org/abs/1711.06232) [find](https://scholar.google.com/scholar?q=Jia%2DHong+Huang%2C+Cuong+Duc+Dao%2C+Modar+Alfadly%2C+and+Bernard+Ghanem%2E+2017%2E+A+novel+framework+for+robustness+analysis+of+visual+qa+models%2E+arXiv+preprint+arXiv%3A1711%2E06232%2E+%E2%80%94+%5BarXiv%3A1711%2E06232%5D%28https%3A%2F%2Farx)
11. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017.
12. Kushal Kafle and Christopher Kanan. 2017. An analysis of visual question answering algorithms. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1983–1991. IEEE. [find](https://scholar.google.com/scholar?q=Kushal+Kafle+and+Christopher+Kanan%2E+2017%2E+An+analysis+of+visual+question+answering+algorithms%2E+In+2017+IEEE+International+Conference+on+Computer+Vision+%28ICCV%29%2C+pages+1983%E2%80%931991%2E+IEEE%2E)
13. Vahid Kazemi and Ali Elqursh. 2017. Show, ask, attend, and answer: A strong baseline for visual question answering. arXiv preprint arXiv:1704.03162. — [arXiv:1704.03162](https://arxiv.org/abs/1704.03162) [find](https://scholar.google.com/scholar?q=Vahid+Kazemi+and+Ali+Elqursh%2E+2017%2E+Show%2C+ask%2C+attend%2C+and+answer%3A+A+strong+baseline+for+visual+question+answering%2E+arXiv+preprint+arXiv%3A1704%2E03162%2E+%E2%80%94+%5BarXiv%3A1704%2E03162%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1704%2E031)
14. Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516–1526. [find](https://scholar.google.com/scholar?q=Jayant+Krishnamurthy%2C+Pradeep+Dasigi%2C+and+Matt+Gardner%2E+2017%2E+Neural+semantic+parsing+with+type+constraints+for+semi%2Dstructured+tables%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Na)
15. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European conference on computer vision, pages 740–755. Springer.
16. Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018. It was the training data pruning too! arXiv preprint arXiv:1803.04579. — [arXiv:1803.04579](https://arxiv.org/abs/1803.04579) [find](https://scholar.google.com/scholar?q=Pramod+Kaushik+Mudrakarta%2C+Ankur+Taly%2C+Mukund+Sundararajan%2C+and+Kedar+Dhamdhere%2E+2018%2E+It+was+the+training+data+pruning+too%21+arXiv+preprint+arXiv%3A1803%2E04579%2E+%E2%80%94+%5BarXiv%3A1803%2E04579%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
17. Arvind Neelakantan, Quoc V. Le, Martı́n Abadi, Andrew McCallum, and Dario Amodei. 2017. Learning a natural language interface with neural programmer. [find](https://scholar.google.com/scholar?q=Arvind+Neelakantan%2C+Quoc+V%2E+Le%2C+Mart%C4%B1%CC%81n+Abadi%2C+Andrew+McCallum%2C+and+Dario+Amodei%2E+2017%2E+Learning+a+natural+language+interface+with+neural+programmer%2E)
18. Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In International Conference on Learning Representations ICLR. [find](https://scholar.google.com/scholar?q=Arvind+Neelakantan%2C+Quoc+V+Le%2C+and+Ilya+Sutskever%2E+2016%2E+Neural+programmer%3A+Inducing+latent+programs+with+gradient+descent%2E+In+International+Conference+on+Learning+Representations+ICLR%2E)
19. Panupong Pasupat and Percy Liang. 2015. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). In In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
20. Panupong Pasupat and Percy Liang. 2016. Inferring logical forms from denotations. arXiv preprint arXiv:1606.06900. [find](https://scholar.google.com/scholar?q=Panupong+Pasupat+and+Percy+Liang%2E+2016%2E+Inferring+logical+forms+from+denotations%2E+arXiv+preprint+arXiv%3A1606%2E06900%2E)
21. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group…
22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100, 000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392.
23. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016a. Nothing else matters: model-agnostic explanations by identifying prediction invariance. arXiv preprint arXiv:1611.05817. — [arXiv:1611.05817](https://arxiv.org/abs/1611.05817) [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2016a%2E+Nothing+else+matters%3A+model%2Dagnostic+explanations+by+identifying+prediction+invariance%2E+arXiv+preprint+arXiv%3A1611%2E05817%2E+%E2%80%94+%5BarXiv%3A1611%2E05)
24. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016b. [Why should i trust you?: Explaining the predictions of any classifier](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/). In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. ACM.
25. Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. 2016. Not just a black box: [Learning important features through propagating activation differences](https://md.research.elamur.ai/free/en/papers/shrikumar-2017-learning-important-features-through-propagating/). CoRR.
26. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. [Deep inside convolutional networks: Visualising image classification models and saliency maps](https://md.research.elamur.ai/free/en/papers/simonyan-2013-deep-inside-convolutional-networks-visualising/). CoRR.
27. Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. 2014. Striving for simplicity: The all convolutional net. CoRR. [find](https://scholar.google.com/scholar?q=Jost+Tobias+Springenberg%2C+Alexey+Dosovitskiy%2C+Thomas+Brox%2C+and+Martin+A%2E+Riedmiller%2E+2014%2E+Striving+for+simplicity%3A+The+all+convolutional+net%2E+CoRR%2E)
28. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 3319–3328. [find](https://scholar.google.com/scholar?q=Mukund+Sundararajan%2C+Ankur+Taly%2C+and+Qiqi+Yan%2E+2017%2E+Axiomatic+attribution+for+deep+networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2C+ICML+2017%2C+Sydney%2C+NSW%2C+Australia)
29. Damien Teney, Peter Anderson, Xiaodong He, and Anton van den Hengel. 2017. Tips and tricks for visual question answering: Learnings from the 2017 challenge. arXiv preprint arXiv:1708.02711. [find](https://scholar.google.com/scholar?q=Damien+Teney%2C+Peter+Anderson%2C+Xiaodong+He%2C+and+Anton+van+den+Hengel%2E+2017%2E+Tips+and+tricks+for+visual+question+answering%3A+Learnings+from+the+2017+challenge%2E+arXiv+preprint+arXiv%3A1708%2E02711%2E)
30. Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Adams+Wei+Yu%2C+David+Dohan%2C+Quoc+Le%2C+Thang+Luong%2C+Rui+Zhao%2C+and+Kai+Chen%2E+2018%2E+Fast+and+accurate+reading+comprehension+by+combining+self%2Dattention+and+convolution%2E+In+International+Conference+on+Learn)
31. Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on, pages 5014–5022. IEEE. [find](https://scholar.google.com/scholar?q=Peng+Zhang%2C+Yash+Goyal%2C+Douglas+Summers%2DStay%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+2016%2E+Yin+and+yang%3A+Balancing+and+answering+binary+visual+questions%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2016)
32. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. [Visual7w: Grounded question answering in images](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4995–5004.

*Updated 24 October 2026*