---
title: "Exploring Models and Data for Image Question Answering (2015)"
description: "The paper addresses image question answering: given a picture and a short natural-language… 32 references, cited by 96 papers here."
authors: ["Mengye Ren", "Ryan Kiros", "Richard Zemel"]
year: 2015
venue: "arXiv:1505.02074"
source: "https://arxiv.org/abs/1505.02074"
doi: "10.48550/arxiv.1505.02074"
arxiv: "1505.02074"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ren-2015-exploring-models-data-image-question/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ren-2015-exploring-models-data-image-question/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / Exploring Models and Data for Image Question Answering

# Exploring Models and Data for Image Question Answering

**Authors:** Mengye Ren, [Ryan Kiros](https://md.research.elamur.ai/free/en/authors/ryan-kiros/) and [Richard Zemel](https://md.research.elamur.ai/free/en/authors/richard-s-zemel/) · **Year:** 2015 · **Venue:** arXiv:1505.02074 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1505.02074) · [PDF](https://arxiv.org/pdf/1505.02074) · [DOI 10.48550/arxiv.1505.02074](https://doi.org/10.48550/arxiv.1505.02074) · [arXiv 1505.02074](https://arxiv.org/abs/1505.02074) · [OpenAlex](https://openalex.org/W1575833922)

**Citations:** 359 citations (OpenAlex, 2026-10-14) · [96](https://md.research.elamur.ai/free/en/papers/ren-2015-exploring-models-data-image-question/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses image question answering: given a picture and a short natural-language question, the system must produce a correct answer. The authors restrict answers to a single word, which turns the task into classification and makes evaluation simpler and more reliable. Their main model, VIS+LSTM, takes CNN features from a frozen 19-layer VGG network, projects them linearly into the word-embedding space, and feeds them to an LSTM as though the image were an extra word in the question. A softmax over candidate answers follows the LSTM. Variants add a backward LSTM or place the image at both ends of the question. They compare this model against simpler baselines such as IMG+BOW and question-only ("blind") models. Because the existing DAQUAR benchmark is small and skewed, they also built a parser-based algorithm that rewrites MS-COCO captions into questions. The result is COCO-QA, a much larger dataset with a more balanced answer distribution. Their models beat prior published DAQUAR results by roughly 1.8×. Notably, a plain bag-of-words question encoder performed about as well as the recurrent one.

## Key points
- DAQUAR has about 1,500 indoor images and 7,000 questions. It comes in 37-class and 894-class variants, and guessing the most frequent answer already scores well on it.
- COCO-QA covers four question types: object, number, color and location. Its training split has 78,736 questions, of which 54,992 are object questions, 5,885 number, 13,059 color and 4,800 location.
- The question generator works as follows:
  - It parses captions with the Stanford parser and splits compound sentences.
  - It swaps "a(n)" for "the" and applies wh-movement under simple constraints such as A-over-A.
  - It uses WordNet and NLTK to identify candidate answers.
- Filtering out answers that were too rare or too frequent reduced the share of the most common answer in the COCO-QA test set from 24.98% to 7.30%.
- The full model improved accuracy by about 50% over the image-only baseline. The models handled color questions well, but counting remained weak and failed when several object types appeared in one image.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 110 works · 179,062 citations
- [Canadian Institute for Advanced Research](https://md.research.elamur.ai/free/en/labs/canadian-institute-for-advanced-research/) — Research institute, Toronto, Canada · 35 works · 19,316 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 487 works

## Cite

Cite as: Mengye Ren, Ryan Kiros and Richard Zemel. 2015. Exploring Models and Data for Image Question Answering. arXiv:1505.02074.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ren2015exploring,
  title = {{Exploring Models and Data for Image Question Answering}},
  author = {Mengye Ren, Ryan Kiros and Richard Zemel},
  year = {2015},
  doi = {10.48550/arxiv.1505.02074},
  eprint = {1505.02074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1505.02074}
}
```

- [← previous Bilingual Word Representations with Monolingual Quality in Mind](https://md.research.elamur.ai/free/en/papers/luong-2015-bilingual-word-representations-monolingual-quality/)
- [next → Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/he-2015-multi-perspective-sentence-similarity-modeling/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “[Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/),” in CVPR, 2015.
2. R. Kiros, R. Salakhutdinov, and R. S. Zemel, “[Unifying visual-semantic embeddings with multimodal neural language models](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/),” TACL, 2015. COCOQA 14855 Where are the ripe bananas sitting? Ground truth: basket IMG+BOW: basket (0.97) 2-VIS+BLSTM: basket (0.58) BOW: bowl (0.48) DAQUAR 585 What is the objec…
3. A. Karpathy, A. Joulin, and L. Fei-Fei, “Deep fragment embeddings for bidirectional image sentence mapping,” in NIPS, 2013. [find](https://scholar.google.com/scholar?q=A%2E+Karpathy%2C+A%2E+Joulin%2C+and+L%2E+Fei%2DFei%2C+%E2%80%9CDeep+fragment+embeddings+for+bidirectional+image+sentence+mapping%2C%E2%80%9D+in+NIPS%2C+2013%2E)
4. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille, “Explain images with multimodal recurrent neural networks,” NIPS Deep Learning Workshop, 2014. [find](https://scholar.google.com/scholar?q=J%2E+Mao%2C+W%2E+Xu%2C+Y%2E+Yang%2C+J%2E+Wang%2C+and+A%2E+L%2E+Yuille%2C+%E2%80%9CExplain+images+with+multimodal+recurrent+neural+networks%2C%E2%80%9D+NIPS+Deep+Learning+Workshop%2C+2014%2E)
5. J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “[Long-term recurrent convolutional networks for visual recognition and description](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/),” in CVPR, 2014.
6. X. Chen and C. L. Zitnick, “Learning a recurrent visual representation for image caption generation,” CoRR, vol. abs/1411.5654, 2014. [find](https://scholar.google.com/scholar?q=X%2E+Chen+and+C%2E+L%2E+Zitnick%2C+%E2%80%9CLearning+a+recurrent+visual+representation+for+image+caption+generation%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1411%2E5654%2C+2014%2E)
7. H. Fang, S. Gupta, F. N. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig, “[From captions to visual concepts and back](https://md.research.elamur.ai/free/en/papers/fang-2014-captions-visual-concepts-back/),” in CVPR, 2015.
8. K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio, “[Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/),” in ICML, 2015.
9. R. Lebret, P. O. Pinheiro, and R. Collobert, “Phrase-based image captioning,” in ICML, 2015. [find](https://scholar.google.com/scholar?q=R%2E+Lebret%2C+P%2E+O%2E+Pinheiro%2C+and+R%2E+Collobert%2C+%E2%80%9CPhrase%2Dbased+image+captioning%2C%E2%80%9D+in+ICML%2C+2015%2E)
10. B. Klein, G. Lev, G. Lev, and L. Wolf, “Fisher vectors derived from hybrid Gaussian-Laplacian mixture models for image annotations,” in CVPR, 2015. [find](https://scholar.google.com/scholar?q=B%2E+Klein%2C+G%2E+Lev%2C+G%2E+Lev%2C+and+L%2E+Wolf%2C+%E2%80%9CFisher+vectors+derived+from+hybrid+Gaussian%2DLaplacian+mixture+models+for+image+annotations%2C%E2%80%9D+in+CVPR%2C+2015%2E)
11. M. Malinowski and M. Fritz, “Towards a visual Turing challenge,” in NIPS Workshop on Learning Semantics, 2014. [find](https://scholar.google.com/scholar?q=M%2E+Malinowski+and+M%2E+Fritz%2C+%E2%80%9CTowards+a+visual+Turing+challenge%2C%E2%80%9D+in+NIPS+Workshop+on+Learning+Semantics%2C+2014%2E)
12. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in ECCV, 2012. [find](https://scholar.google.com/scholar?q=N%2E+Silberman%2C+D%2E+Hoiem%2C+P%2E+Kohli%2C+and+R%2E+Fergus%2C+%E2%80%9CIndoor+segmentation+and+support+inference+from+RGBD+images%2C%E2%80%9D+in+ECCV%2C+2012%2E)
13. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “VQA: Visual Question Answering,” CoRR, vol. abs/1505.00468, 2015. [find](https://scholar.google.com/scholar?q=S%2E+Antol%2C+A%2E+Agrawal%2C+J%2E+Lu%2C+M%2E+Mitchell%2C+D%2E+Batra%2C+C%2E+L%2E+Zitnick%2C+and+D%2E+Parikh%2C+%E2%80%9CVQA%3A+Visual+Question+Answering%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1505%2E00468%2C+2015%2E)
14. M. Malinowski, M. Rohrbach, and M. Fritz, “Ask Your Neurons: A Neural-based Approach to Answering Questions about Images,” CoRR, vol. abs/1505.01121, 2015. — [arXiv:1505.01121](https://arxiv.org/abs/1505.01121) [find](https://scholar.google.com/scholar?q=M%2E+Malinowski%2C+M%2E+Rohrbach%2C+and+M%2E+Fritz%2C+%E2%80%9CAsk+Your+Neurons%3A+A+Neural%2Dbased+Approach+to+Answering+Questions+about+Images%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1505%2E01121%2C+2015%2E+%E2%80%94+%5BarXiv%3A1505%2E01121%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
15. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu, “Are you talking to a machine? dataset and methods for multilingual image question answering,” CoRR, vol. abs/1505.05612, 2015. [find](https://scholar.google.com/scholar?q=H%2E+Gao%2C+J%2E+Mao%2C+J%2E+Zhou%2C+Z%2E+Huang%2C+L%2E+Wang%2C+and+W%2E+Xu%2C+%E2%80%9CAre+you+talking+to+a+machine%3F+dataset+and+methods+for+multilingual+image+question+answering%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1505%2E05612%2C+2015%2E)
16. L. Ma, Z. Lu, and H. Li, “Learning to answer questions from image using convolutional neural network,” CoRR, vol. abs/1506.00333, 2015. [find](https://scholar.google.com/scholar?q=L%2E+Ma%2C+Z%2E+Lu%2C+and+H%2E+Li%2C+%E2%80%9CLearning+to+answer+questions+from+image+using+convolutional+neural+network%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1506%2E00333%2C+2015%2E)
17. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “[Microsoft COCO: Common Objects in Context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/),” in ECCV, 2014.
18. X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick, “[Microsoft COCO captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/),” CoRR, vol. abs/1504.00325, 2015.
19. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2C+%E2%80%9CLong+short%2Dterm+memory%2C%E2%80%9D+Neural+Computation%2C+vol%2E+9%2C+no%2E+8%2C+pp%2E+1735%E2%80%931780%2C+1997%2E)
20. K. Simonyan and A. Zisserman, “[Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/),” in ICLR, 2015.
21. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei, “[Imagenet large scale visual recognition challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/),” IJCV, 2015.
22. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “[Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/),” in ICLR, 2013.
23. A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “DeViSE: A deep visual-semantic embedding model,” in NIPS, 2013. [find](https://scholar.google.com/scholar?q=A%2E+Frome%2C+G%2E+S%2E+Corrado%2C+J%2E+Shlens%2C+S%2E+Bengio%2C+J%2E+Dean%2C+M%2E+Ranzato%2C+and+T%2E+Mikolov%2C+%E2%80%9CDeViSE%3A+A+deep+visual%2Dsemantic+embedding+model%2C%E2%80%9D+in+NIPS%2C+2013%2E)
24. M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,” J. Artif. Intell. Res. (JAIR), vol. 47, pp. 853–899, 2013. [find](https://scholar.google.com/scholar?q=M%2E+Hodosh%2C+P%2E+Young%2C+and+J%2E+Hockenmaier%2C+%E2%80%9CFraming+image+description+as+a+ranking+task%3A+Data%2C+models+and+evaluation+metrics%2C%E2%80%9D+J%2E+Artif%2E+Intell%2E+Res%2E+%28JAIR%29%2C+vol%2E+47%2C+pp%2E+853%E2%80%93899%2C+2013%2E)
25. V. Ordonez, G. Kulkarni, and T. L. Berg, “Im2text: Describing images using 1 million captioned photographs,” in NIPS, 2011. [find](https://scholar.google.com/scholar?q=V%2E+Ordonez%2C+G%2E+Kulkarni%2C+and+T%2E+L%2E+Berg%2C+%E2%80%9CIm2text%3A+Describing+images+using+1+million+captioned+photographs%2C%E2%80%9D+in+NIPS%2C+2011%2E)
26. D. Klein and C. D. Manning, “[Accurate unlexicalized parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/),” in ACL, 2003.
27. N. Chomsky, Conditions on Transformations. New York: Academic Press, 1973. [find](https://scholar.google.com/scholar?q=N%2E+Chomsky%2C+Conditions+on+Transformations%2E+New+York%3A+Academic+Press%2C+1973%2E)
28. C. Fellbaum, Ed., WordNet An Electronic Lexical Database. Cambridge, MA; London: The MIT Press, May 1998. [find](https://scholar.google.com/scholar?q=C%2E+Fellbaum%2C+Ed%2E%2C+WordNet+An+Electronic+Lexical+Database%2E+Cambridge%2C+MA%3B+London%3A+The+MIT+Press%2C+May+1998%2E)
29. S. Bird, “[NLTK: the natural language toolkit](https://md.research.elamur.ai/free/en/papers/bird-2006-nltk-natural-language-toolkit/),” in ACL, 2006.
30. J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick, “Exploring nearest neighbor approaches for image captioning,” CoRR, vol. abs/1505.04467, 2015. [find](https://scholar.google.com/scholar?q=J%2E+Devlin%2C+S%2E+Gupta%2C+R%2E+Girshick%2C+M%2E+Mitchell%2C+and+C%2E+L%2E+Zitnick%2C+%E2%80%9CExploring+nearest+neighbor+approaches+for+image+captioning%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1505%2E04467%2C+2015%2E)
31. Z. Wu and M. Palmer, “[Verb semantics and lexical selection](https://md.research.elamur.ai/free/en/papers/wu-1994-verb-semantics-lexical-selection/),” in ACL, 1994.
32. M. Malinowski and M. Fritz, “A multi-world approach to question answering about real-world scenes based on uncertain input,” in NIPS, 2014. [find](https://scholar.google.com/scholar?q=M%2E+Malinowski+and+M%2E+Fritz%2C+%E2%80%9CA+multi%2Dworld+approach+to+question+answering+about+real%2Dworld+scenes+based+on+uncertain+input%2C%E2%80%9D+in+NIPS%2C+2014%2E)

*Updated 2 November 2026*