---
title: "Stacked Attention Networks for Image Question Answering (2015)"
description: "That design struggles when the answer depends on small, specific parts of the scene. 4.8 points. 32 references, cited by 159 papers here."
authors: ["Zichao Yang", "Xiaodong He", "Jianfeng Gao", "Li Deng", "Alex Smola"]
year: 2015
venue: "arXiv:1511.02274"
source: "https://arxiv.org/abs/1511.02274"
doi: "10.48550/arxiv.1511.02274"
arxiv: "1511.02274"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2015-stacked-attention-networks-image-question/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2015-stacked-attention-networks-image-question/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Stacked Attention Networks for Image Question Answering

# Stacked Attention Networks for Image Question Answering

**Authors:** [Zichao Yang](https://md.research.elamur.ai/free/en/authors/zichao-yang/), [Xiaodong He](https://md.research.elamur.ai/free/en/authors/xiaodong-he/), [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/), [Li Deng](https://md.research.elamur.ai/free/en/authors/li-deng/) and [Alex Smola](https://md.research.elamur.ai/free/en/authors/alex-j-smola/) · **Year:** 2015 · **Venue:** [arXiv:1511.02274](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/) ([IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2015](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/2015/)) · **Publisher:** [Institute of Electrical and Electronics Engineers](https://md.research.elamur.ai/free/en/publishers/institute-of-electrical-and-electronics-engineers/) · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1511.02274) · [PDF](https://arxiv.org/pdf/1511.02274) · [DOI 10.48550/arxiv.1511.02274](https://doi.org/10.48550/arxiv.1511.02274) · [arXiv 1511.02274](https://arxiv.org/abs/1511.02274) · [OpenAlex](https://openalex.org/W2963954913)

**Citations:** 1,998 citations (OpenAlex, 2026-10-14) · [159](https://md.research.elamur.ai/free/en/papers/yang-2015-stacked-attention-networks-image-question/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses image question answering (QA): answering a natural-language question about a given picture. Earlier neural approaches merged a single global CNN image vector with a question encoding. That design struggles when the answer depends on small, specific parts of the scene. The authors argue that such questions need several reasoning steps, so they introduce stacked attention networks (SANs). VGGNet features from the last pooling layer split each image into a 14×14 grid of region vectors. An LSTM or a CNN with unigram, bigram and trigram filters encodes the question. The question vector then serves as a query that weights the image regions. The attended visual summary is added to the query, producing a refined query for the next attention layer, so the focus narrows step by step. SANs beat prior systems on four image QA benchmarks, including COCO-QA and VQA. Attention maps show the first layer spreading over the objects the question mentions and the second layer concentrating on the region holding the answer. The work showed that iterative visual attention is effective for multimodal reasoning.

## Key points
- Images are resized to 448×448. The last VGGNet pooling layer then yields 512×14×14 features, one vector per 32×32-pixel region, which a tanh layer projects to the question-vector dimension.
- On the official VQA server, SAN(2, CNN) reaches 58.7% overall (79.3 Yes/No, 36.6 Number, 46.1 Other). This is 4.8 points above the strongest baseline, LSTM Q+I, with the largest gain (9.7 points) on the Other question type.
- On COCO-QA, SAN(2, CNN) improves over the IMG+BOW and 2-VIS+BLSTM baselines by 7.2 points on Color, 6.1 on Objects, 5.7 on Location and 4.2 on Number.
- Two attention layers beat one on all four data sets. Yes/No questions benefit little, since their answers depend mostly on the question text.
- A manual check of 100 COCO-QA errors found these causes:
  - 22%: attention on the wrong region;
  - 42%: correct region but wrong answer;
  - 31%: ambiguous but plausible answers;
  - 5%: incorrect gold labels.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations
- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

💡 Concepts (1): [💡 Question answering](https://md.research.elamur.ai/free/en/concepts/question-answering/) · 53 works

## Cite

Cite as: Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng and Alex Smola. 2015. Stacked Attention Networks for Image Question Answering. arXiv:1511.02274.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yang2015stacked,
  title = {{Stacked Attention Networks for Image Question Answering}},
  author = {Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng and Alex Smola},
  year = {2015},
  doi = {10.48550/arxiv.1511.02274},
  eprint = {1511.02274},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.02274}
}
```

- [← previous Named Entity Recognition with Bidirectional LSTM-CNNs](https://md.research.elamur.ai/free/en/papers/chiu-2015-named-entity-recognition-bidirectional-lstm/)
- [next → chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. arXiv preprint arXiv:1505.00468, 2015. 1, 2, 5, 6, 7 [find](https://scholar.google.com/scholar?q=S%2E+Antol%2C+A%2E+Agrawal%2C+J%2E+Lu%2C+M%2E+Mitchell%2C+D%2E+Batra%2C+C%2E+L%2E+Zitnick%2C+and+D%2E+Parikh%2E+Vqa%3A+Visual+question+answering%2E+arXiv+preprint+arXiv%3A1505%2E00468%2C+2015%2E+1%2C+2%2C+5%2C+6%2C+7)
2. D. Bahdanau, K. Cho, and Y. Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014. 1
3. J. Berant and P. Liang. [Semantic parsing via paraphrasing](https://md.research.elamur.ai/free/en/papers/berant-2014-semantic-parsing-paraphrasing/). In Proceedings of ACL, volume 7, page 92, 2014. 1
4. A. Bordes, S. Chopra, and J. Weston. Question answering with subgraph embeddings. arXiv preprint arXiv:1406.3676, 2014. 1 [find](https://scholar.google.com/scholar?q=A%2E+Bordes%2C+S%2E+Chopra%2C+and+J%2E+Weston%2E+Question+answering+with+subgraph+embeddings%2E+arXiv+preprint+arXiv%3A1406%2E3676%2C+2014%2E+1)
5. X. Chen and C. L. Zitnick. Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654, 2014. 2 [find](https://scholar.google.com/scholar?q=X%2E+Chen+and+C%2E+L%2E+Zitnick%2E+Learning+a+recurrent+visual+representation+for+image+caption+generation%2E+arXiv+preprint+arXiv%3A1411%2E5654%2C+2014%2E+2)
6. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt, et al. From captions to visual concepts and back. arXiv preprint arXiv:1411.4952, 2014. 2, 3, 5 [find](https://scholar.google.com/scholar?q=H%2E+Fang%2C+S%2E+Gupta%2C+F%2E+Iandola%2C+R%2E+Srivastava%2C+L%2E+Deng%2C+P%2E+Dolla%CC%81r%2C+J%2E+Gao%2C+X%2E+He%2C+M%2E+Mitchell%2C+J%2E+Platt%2C+et+al%2E+From+captions+to+visual+concepts+and+back%2E+arXiv+preprint+arXiv%3A1411%2E4952%2C+2014%2E+2%2C+3%2C+5)
7. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. arXiv preprint arXiv:1505.05612, 2015. 1, 2 [find](https://scholar.google.com/scholar?q=H%2E+Gao%2C+J%2E+Mao%2C+J%2E+Zhou%2C+Z%2E+Huang%2C+L%2E+Wang%2C+and+W%2E+Xu%2E+Are+you+talking+to+a+machine%3F+dataset+and+methods+for+multilingual+image+question+answering%2E+arXiv+preprint+arXiv%3A1505%2E05612%2C+2015%2E+1%2C+2)
8. A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. 5 [find](https://scholar.google.com/scholar?q=A%2E+Graves%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1308%2E0850%2C+2013%2E+5)
9. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 1 [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E+1)
10. A. Karpathy and L. Fei-Fei. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). arXiv preprint arXiv:1412.2306, 2014. 2
11. Y. Kim. [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). arXiv preprint arXiv:1408.5882, 2014. 3
12. R. Kiros, R. Salakhutdinov, and R. S. Zemel. [Unifying visual-semantic embeddings with multimodal neural language models](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/). arXiv preprint arXiv:1411.2539, 2014. 2
13. A. Krizhevsky, I. Sutskever, and G. E. Hinton. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In Advances in neural information processing systems, pages 1097–1105, 2012. 2
14. A. Kumar, O. Irsoy, J. Su, J. Bradbury, R. English, B. Pierce, P. Ondruska, I. Gulrajani, and R. Socher. [Ask me anything: Dynamic memory networks for natural language processing](https://md.research.elamur.ai/free/en/papers/kumar-2015-ask-me-anything-dynamic-memory/). arXiv preprint arXiv:1506.07285, 2015. 1
15. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 1 [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+L%2E+Bottou%2C+Y%2E+Bengio%2C+and+P%2E+Haffner%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E+1)
16. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision–ECCV 2014, pages 740–755. Springer, 2014. 5
17. L. Ma, Z. Lu, and H. Li. Learning to answer questions from image using convolutional neural network. arXiv preprint arXiv:1506.00333, 2015. 2, 5, 6 [find](https://scholar.google.com/scholar?q=L%2E+Ma%2C+Z%2E+Lu%2C+and+H%2E+Li%2E+Learning+to+answer+questions+from+image+using+convolutional+neural+network%2E+arXiv+preprint+arXiv%3A1506%2E00333%2C+2015%2E+2%2C+5%2C+6)
18. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in Neural Information Processing Systems, pages 1682–1690, 2014. 1, 2, 4, 5, 6 [find](https://scholar.google.com/scholar?q=M%2E+Malinowski+and+M%2E+Fritz%2E+A+multi%2Dworld+approach+to+question+answering+about+real%2Dworld+scenes+based+on+uncertain+input%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+1682%E2%80%931690%2C+2014%2E+)
19. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. arXiv preprint arXiv:1505.01121, 2015. 1, 2, 5, 6 — [arXiv:1505.01121](https://arxiv.org/abs/1505.01121) [find](https://scholar.google.com/scholar?q=M%2E+Malinowski%2C+M%2E+Rohrbach%2C+and+M%2E+Fritz%2E+Ask+your+neurons%3A+A+neural%2Dbased+approach+to+answering+questions+about+images%2E+arXiv+preprint+arXiv%3A1505%2E01121%2C+2015%2E+1%2C+2%2C+5%2C+6+%E2%80%94+%5BarXiv%3A1505%2E01121%5D%28https%3A%2F%2F)
20. J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632, 2014. 2 [find](https://scholar.google.com/scholar?q=J%2E+Mao%2C+W%2E+Xu%2C+Y%2E+Yang%2C+J%2E+Wang%2C+and+A%2E+Yuille%2E+Deep+captioning+with+multimodal+recurrent+neural+networks+%28m%2Drnn%29%2E+arXiv+preprint+arXiv%3A1412%2E6632%2C+2014%2E+2)
21. M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. arXiv preprint arXiv:1505.02074, 2015. 1, 2, 5, 6 [find](https://scholar.google.com/scholar?q=M%2E+Ren%2C+R%2E+Kiros%2C+and+R%2E+Zemel%2E+Exploring+models+and+data+for+image+question+answering%2E+arXiv+preprint+arXiv%3A1505%2E02074%2C+2015%2E+1%2C+2%2C+5%2C+6)
22. Y. Shen, X. He, J. Gao, L. Deng, and G. Mesnil. A latent semantic model with convolutional-pooling structure for information retrieval. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 101–110. ACM, 2014. 3 [find](https://scholar.google.com/scholar?q=Y%2E+Shen%2C+X%2E+He%2C+J%2E+Gao%2C+L%2E+Deng%2C+and+G%2E+Mesnil%2E+A+latent+semantic+model+with+convolutional%2Dpooling+structure+for+information+retrieval%2E+In+Proceedings+of+the+23rd+ACM+International+Conference+on+Confe)
23. K. Simonyan and A. Zisserman. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). arXiv preprint arXiv:1409.1556, 2014. 2
24. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014. 5 [find](https://scholar.google.com/scholar?q=N%2E+Srivastava%2C+G%2E+Hinton%2C+A%2E+Krizhevsky%2C+I%2E+Sutskever%2C+and+R%2E+Salakhutdinov%2E+Dropout%3A+A+simple+way+to+prevent+neural+networks+from+overfitting%2E+The+Journal+of+Machine+Learning+Research%2C+15%281%29%3A1929%E2%80%93195)
25. I. Sutskever, O. Vinyals, and Q. V. Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in neural information processing systems, pages 3104–3112, 2014. 3
26. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. 2 [find](https://scholar.google.com/scholar?q=C%2E+Szegedy%2C+W%2E+Liu%2C+Y%2E+Jia%2C+P%2E+Sermanet%2C+S%2E+Reed%2C+D%2E+Anguelov%2C+D%2E+Erhan%2C+V%2E+Vanhoucke%2C+and+A%2E+Rabinovich%2E+Going+deeper+with+convolutions%2E+arXiv+preprint+arXiv%3A1409%2E4842%2C+2014%2E+2)
27. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). arXiv preprint arXiv:1411.4555, 2014. 2
28. J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014. 1 [find](https://scholar.google.com/scholar?q=J%2E+Weston%2C+S%2E+Chopra%2C+and+A%2E+Bordes%2E+Memory+networks%2E+arXiv+preprint+arXiv%3A1410%2E3916%2C+2014%2E+1)
29. Z. Wu and M. Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 133–138. Association for Computational Linguistics, 1994. 5 [find](https://scholar.google.com/scholar?q=Z%2E+Wu+and+M%2E+Palmer%2E+Verbs+semantics+and+lexical+selection%2E+In+Proceedings+of+the+32nd+annual+meeting+on+Association+for+Computational+Linguistics%2C+pages+133%E2%80%93138%2E+Association+for+Computational+Linguis)
30. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. [Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). arXiv preprint arXiv:1502.03044, 2015. 1, 2
31. W.-t. Yih, M.-W. Chang, X. He, and J. Gao. [Semantic parsing via staged query graph generation: Question answering with knowledge base](https://md.research.elamur.ai/free/en/papers/yih-2015-semantic-parsing-staged-query-graph/). In Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP,…
32. W.-t. Yih, X. He, and C. Meek. [Semantic parsing for single-relation question answering](https://md.research.elamur.ai/free/en/papers/yih-2014-semantic-parsing-single-relation-question/). In Proceedings of ACL, 2014. 1 Figure 7: More examples

*Updated 30 October 2026*