---
title: "The Neuro-Symbolic Concept Learner: Interpreting Scenes,"
description: "This paper presents the Neuro-Symbolic Concept Learner (NS-CL), which needs only images paired… 52 references, cited by 113 papers here."
authors: ["Jiayuan Mao", "Chuang Gan", "Pushmeet Kohli", "Joshua B. Tenenbaum", "Jiajun Wu"]
year: 2019
venue: "arXiv:1904.12584"
source: "https://arxiv.org/abs/1904.12584"
doi: "10.48550/arxiv.1904.12584"
arxiv: "1904.12584"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mao-2019-neuro-symbolic-concept-learner-interpreting/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mao-2019-neuro-symbolic-concept-learner-interpreting/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

# The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

**Authors:** [Jiayuan Mao](https://md.research.elamur.ai/free/en/authors/jiayuan-mao/), [Chuang Gan](https://md.research.elamur.ai/free/en/authors/chuang-gan/), [Pushmeet Kohli](https://md.research.elamur.ai/free/en/authors/pushmeet-kohli/), [Joshua B. Tenenbaum](https://md.research.elamur.ai/free/en/authors/joshua-b-tenenbaum/) and [Jiajun Wu](https://md.research.elamur.ai/free/en/authors/jiajun-wu/) · **Year:** 2019 · **Venue:** arXiv:1904.12584 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1904.12584) · [PDF](https://arxiv.org/pdf/1904.12584) · [DOI 10.48550/arxiv.1904.12584](https://doi.org/10.48550/arxiv.1904.12584) · [arXiv 1904.12584](https://arxiv.org/abs/1904.12584) · [OpenAlex](https://openalex.org/W2908791737)

**Citations:** 128 citations (OpenAlex, 2026-10-14) · [113](https://md.research.elamur.ai/free/en/papers/mao-2019-neuro-symbolic-concept-learner-interpreting/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Visual reasoning systems that run explicit programs are usually trained with costly extra annotations, such as ground-truth programs or labelled object attributes. This paper presents the Neuro-Symbolic Concept Learner (NS-CL), which needs only images paired with questions and answers. NS-CL has three parts. A perception module builds a latent vector for each detected object. A semantic parser turns each question into a program in a domain-specific language. A quasi-symbolic executor runs that program over the object representations. The executor is deterministic but differentiable, so the visual features and concept embeddings learn from answer correctness. The parser, whose output is discrete, is trained with REINFORCE. Training follows a curriculum that moves from small scenes and short questions to crowded scenes and compositional queries. NS-CL reaches state-of-the-art accuracy on CLEVR without any program annotations. It also generalizes to larger scenes, longer programs, new attribute combinations (CLEVR-CoGenT) and new concepts, and it transfers to image-caption retrieval. Because every reasoning step is explicit, errors can be traced and ambiguous questions detected. This makes the approach a useful template for interpretable, weakly supervised grounding of language.

## Key points
- **Architecture:** A pretrained Mask R-CNN proposes objects, and ResNet-based features represent each one. Attributes such as Shape and Color act as neural operators that map objects into a joint visual-semantic space, where classification uses similarity to concept embeddings.
- **Supervision compared with prior models:** IEP and TbD rely on 700K program annotations, and NS-VQA uses 0.2K programs plus attribute labels. NS-CL uses zero programs and no attribute labels.
- **Curriculum:** Training starts with scenes of at most 3 objects and simple attribute questions. It then adds relational questions (e.g., "right of", same material) and finally complex, highly compositional queries.
- **Evaluation:** Experiments cover CLEVR, CLEVR-CoGenT, a Minecraft reasoning dataset and natural images from VQS. On VQS, NS-CL performs comparably to an MLP baseline and the MAC network.
- **Interpretability:** Execution traces show where reasoning fails, for example a misread material or a missed detection. The executor also flags references that are ambiguous or point to objects that do not exist.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum and Jiajun Wu. 2019. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. arXiv:1904.12584.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mao2019neuro,
  title = {{The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision}},
  author = {Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum and Jiajun Wu},
  year = {2019},
  doi = {10.48550/arxiv.1904.12584},
  eprint = {1904.12584},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.12584}
}
```

- [← previous Training Neural Machine Translation to Apply Terminology Constraints](https://md.research.elamur.ai/free/en/papers/dinu-2019-training-neural-machine-translation-apply/)
- [next → Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling](https://md.research.elamur.ai/free/en/papers/han-2019-unsupervised-domain-adaptation-contextualized-embeddings/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

52 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend, Tom Kwiatkowski, Nathaniel J Smith, Sharon Goldwater, and Mark Steedman. Bootstrapping language acquisition. Cognition, 2017. [find](https://scholar.google.com/scholar?q=Omri+Abend%2C+Tom+Kwiatkowski%2C+Nathaniel+J+Smith%2C+Sharon+Goldwater%2C+and+Mark+Steedman%2E+Bootstrapping+language+acquisition%2E+Cognition%2C+2017%2E)
2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Xiaodong+He%2C+Chris+Buehler%2C+Damien+Teney%2C+Mark+Johnson%2C+Stephen+Gould%2C+and+Lei+Zhang%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In+CVPR%2C+2018)
3. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. [Learning to compose neural networks for question answering](https://md.research.elamur.ai/free/en/papers/andreas-2016-learning-compose-neural-networks-question/). In NAACL-HLT, 2016.
4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV, 2015. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+VQA%3A+Visual+question+answering%2E+In+ICCV%2C+2015%2E)
5. Yoav Artzi and Luke Zettlemoyer. [Weakly supervised learning of semantic parsers for mapping instructions to actions](https://md.research.elamur.ai/free/en/papers/artzi-2013-weakly-supervised-learning-semantic-parsers/). TACL, 2013.
6. Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In ECCV, 2018. [find](https://scholar.google.com/scholar?q=Fabien+Baradel%2C+Natalia+Neverova%2C+Christian+Wolf%2C+Julien+Mille%2C+and+Greg+Mori%2E+Object+level+visual+reasoning+in+videos%2E+In+ECCV%2C+2018%2E)
7. Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. Abc-cnn: An attention based convolutional neural network for visual question answering. arXiv:1511.05960, 2015. — [arXiv:1511.05960](https://arxiv.org/abs/1511.05960) [find](https://scholar.google.com/scholar?q=Kan+Chen%2C+Jiang+Wang%2C+Liang%2DChieh+Chen%2C+Haoyuan+Gao%2C+Wei+Xu%2C+and+Ram+Nevatia%2E+Abc%2Dcnn%3A+An+attention+based+convolutional+neural+network+for+visual+question+answering%2E+arXiv%3A1511%2E05960%2C+2015%2E+%E2%80%94+%5BarXiv%3A1)
8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. — [Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/)
9. Grzegorz Chrupała, Akos Kádár, and Afra Alishahi. Learning language through pictures. In ACL, 2015. [find](https://scholar.google.com/scholar?q=Grzegorz+Chrupa%C5%82a%2C+Akos+Ka%CC%81da%CC%81r%2C+and+Afra+Alishahi%2E+Learning+language+through+pictures%2E+In+ACL%2C+2015%2E)
10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+CVPR%2C+2009%2E)
11. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. [Long-term recurrent convolutional networks for visual recognition and description](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/). In CVPR, 2015.
12. Li Dong and Mirella Lapata. [Language to logical form with neural attention](https://md.research.elamur.ai/free/en/papers/dong-2016-language-logical-form-neural-attention/). In ACL, 2016.
13. Afsaneh Fazly, Afra Alishahi, and Suzanne Stevenson. A probabilistic computational model of cross-situational word learning. Cognit. Sci., 2010. [find](https://scholar.google.com/scholar?q=Afsaneh+Fazly%2C+Afra+Alishahi%2C+and+Suzanne+Stevenson%2E+A+probabilistic+computational+model+of+cross%2Dsituational+word+learning%2E+Cognit%2E+Sci%2E%2C+2010%2E)
14. Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. VQS: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Chuang+Gan%2C+Yandong+Li%2C+Haoxiang+Li%2C+Chen+Sun%2C+and+Boqing+Gong%2E+VQS%3A+Linking+segmentations+to+questions+and+answers+for+supervised+attention+in+vqa+and+question%2Dfocused+semantic+segmentation%2E+In+ICCV%2C)
15. Siddha Ganju, Olga Russakovsky, and Abhinav Gupta. What’s in a question: Using visual questions as a form of supervision. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Siddha+Ganju%2C+Olga+Russakovsky%2C+and+Abhinav+Gupta%2E+What%E2%80%99s+in+a+question%3A+Using+visual+questions+as+a+form+of+supervision%2E+In+CVPR%2C+2017%2E)
16. Jon Gauthier, Roger Levy, and Joshua B Tenenbaum. Word learning and the acquisition of syntactic– semantic overhypotheses. In CogSci, 2018. [find](https://scholar.google.com/scholar?q=Jon+Gauthier%2C+Roger+Levy%2C+and+Joshua+B+Tenenbaum%2E+Word+learning+and+the+acquisition+of+syntactic%E2%80%93+semantic+overhypotheses%2E+In+CogSci%2C+2018%2E)
17. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In ICML, 2017.
18. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CVPR, 2015.
19. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Georgia+Gkioxari%2C+Piotr+Dolla%CC%81r%2C+and+Ross+Girshick%2E+Mask+R%2DCNN%2E+In+ICCV%2C+2017%2E)
20. Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: learning abstract hierarchical compositional visual concepts. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Irina+Higgins%2C+Nicolas+Sonnerat%2C+Loic+Matthey%2C+Arka+Pal%2C+Christopher+P+Burgess%2C+Matthew+Botvinick%2C+Demis+Hassabis%2C+and+Alexander+Lerchner%2E+Scan%3A+learning+abstract+hierarchical+compositional+visual+con)
21. Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In ECCV, 2018. [find](https://scholar.google.com/scholar?q=Ronghang+Hu%2C+Jacob+Andreas%2C+Trevor+Darrell%2C+and+Kate+Saenko%2E+Explainable+neural+computation+via+stack+neural+module+networks%2E+In+ECCV%2C+2018%2E)
22. Drew A Hudson and Christopher D Manning. [Compositional attention networks for machine reasoning](https://md.research.elamur.ai/free/en/papers/hudson-2018-compositional-attention-networks-machine-reasoning/). In ICLR, 2018.
23. Allan Jabri, Armand Joulin, and Laurens van der Maaten. Revisiting visual question answering baselines. In ECCV, 2016. [find](https://scholar.google.com/scholar?q=Allan+Jabri%2C+Armand+Joulin%2C+and+Laurens+van+der+Maaten%2E+Revisiting+visual+question+answering+baselines%2E+In+ECCV%2C+2016%2E)
24. Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Justin+Johnson%2C+Ranjay+Krishna%2C+Michael+Stark%2C+Li%2DJia+Li%2C+David+Shamma%2C+Michael+Bernstein%2C+and+Li+Fei%2DFei%2E+Image+retrieval+using+scene+graphs%2E+In+CVPR%2C+2015%2E)
25. Justin Johnson, Andrej Karpathy, and Li Fei-Fei. DenseCap: Fully convolutional localization networks for dense captioning. In CVPR, 2016. [find](https://scholar.google.com/scholar?q=Justin+Johnson%2C+Andrej+Karpathy%2C+and+Li+Fei%2DFei%2E+DenseCap%3A+Fully+convolutional+localization+networks+for+dense+captioning%2E+In+CVPR%2C+2016%2E)
26. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. [CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/). In CVPR, 2017a.
27. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. [Inferring and executing programs for visual reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2017-inferring-executing-programs-visual-reasoning/). In ICCV, 2017b.
28. Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. [Unifying visual-semantic embeddings with multimodal neural language models](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/). arXiv:1411.2539, 2014.
29. George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. JAIR, 2018. [find](https://scholar.google.com/scholar?q=George+Konidaris%2C+Leslie+Pack+Kaelbling%2C+and+Tomas+Lozano%2DPerez%2E+From+skills+to+symbols%3A+Learning+symbolic+representations+for+abstract+high%2Dlevel+planning%2E+JAIR%2C+2018%2E)
30. Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In NeurIPS, 2015. [find](https://scholar.google.com/scholar?q=Tejas+D+Kulkarni%2C+William+F+Whitney%2C+Pushmeet+Kohli%2C+and+Josh+Tenenbaum%2E+Deep+convolutional+inverse+graphics+network%2E+In+NeurIPS%2C+2015%2E)
31. Beth Levin. English verb classes and alternations, volume 1. University of Chicago Press, 1993. [find](https://scholar.google.com/scholar?q=Beth+Levin%2E+English+verb+classes+and+alternations%2C+volume+1%2E+University+of+Chicago+Press%2C+1993%2E)
32. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In ECCV, 2014.
33. M Malinowski and M Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NeurIPS, 2014. [find](https://scholar.google.com/scholar?q=M+Malinowski+and+M+Fritz%2E+A+multi%2Dworld+approach+to+question+answering+about+real%2Dworld+scenes+based+on+uncertain+input%2E+In+NeurIPS%2C+2014%2E)
34. Junhua Mao, Jiajing Xu, Kevin Jing, and Alan L Yuille. Training and evaluating multimodal word embeddings with large-scale web annotated images. In NeurIPS, 2016. [find](https://scholar.google.com/scholar?q=Junhua+Mao%2C+Jiajing+Xu%2C+Kevin+Jing%2C+and+Alan+L+Yuille%2E+Training+and+evaluating+multimodal+word+embeddings+with+large%2Dscale+web+annotated+images%2E+In+NeurIPS%2C+2016%2E)
35. David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Closing the gap between performance and interpretability in visual reasoning. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=David+Mascharka%2C+Philip+Tran%2C+Ryan+Soklaski%2C+and+Arjun+Majumdar%2E+Transparency+by+design%3A+Closing+the+gap+between+performance+and+interpretability+in+visual+reasoning%2E+In+CVPR%2C+2018%2E)
36. Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In ICML, 2017. [find](https://scholar.google.com/scholar?q=Junhyuk+Oh%2C+Satinder+Singh%2C+Honglak+Lee%2C+and+Pushmeet+Kohli%2E+Zero%2Dshot+task+generalization+with+multi%2Dtask+deep+reinforcement+learning%2E+In+ICML%2C+2017%2E)
37. Jeffrey Pennington, Richard Socher, and Christopher Manning. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In EMNLP, 2014.
38. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. [Film: Visual reasoning with a general conditioning layer](https://md.research.elamur.ai/free/en/papers/perez-2017-film-visual-reasoning-general-conditioning/). In AAAI, 2018.
39. Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. [A simple neural network module for relational reasoning](https://md.research.elamur.ai/free/en/papers/santoro-2017-simple-neural-network-module-relational/). In NeurIPS, 2017.
40. Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In EMNLP Workshop, 2015. [find](https://scholar.google.com/scholar?q=Sebastian+Schuster%2C+Ranjay+Krishna%2C+Angel+Chang%2C+Li+Fei%2DFei%2C+and+Christopher+D+Manning%2E+Generating+semantically+precise+scene+graphs+from+textual+descriptions+for+improved+image+retrieval%2E+In+EMNLP+Wo)
41. Haoyue Shi, Jiayuan Mao, Tete Xiao, Yuning Jiang, and Jian Sun. Learning visually-grounded semantics from contrastive adversarial samples. In COLING, 2018. [find](https://scholar.google.com/scholar?q=Haoyue+Shi%2C+Jiayuan+Mao%2C+Tete+Xiao%2C+Yuning+Jiang%2C+and+Jian+Sun%2E+Learning+visually%2Dgrounded+semantics+from+contrastive+adversarial+samples%2E+In+COLING%2C+2018%2E)
42. N Siddharth, T. B. Paige, J.W. Meent, A. Desmaison, N. Goodman, P. Kohli, F. Wood, and P. Torr. Learning disentangled representations with semi-supervised deep generative models. In NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=N+Siddharth%2C+T%2E+B%2E+Paige%2C+J%2EW%2E+Meent%2C+A%2E+Desmaison%2C+N%2E+Goodman%2C+P%2E+Kohli%2C+F%2E+Wood%2C+and+P%2E+Torr%2E+Learning+disentangled+representations+with+semi%2Dsupervised+deep+generative+models%2E+In+NeurIPS%2C+2017%2E)
43. Joseph Suarez, Justin Johnson, and Fei-Fei Li. DDRprog: A clevr differentiable dynamic reasoning programmer. arXiv:1803.11361, 2018. — [arXiv:1803.11361](https://arxiv.org/abs/1803.11361) [find](https://scholar.google.com/scholar?q=Joseph+Suarez%2C+Justin+Johnson%2C+and+Fei%2DFei+Li%2E+DDRprog%3A+A+clevr+differentiable+dynamic+reasoning+programmer%2E+arXiv%3A1803%2E11361%2C+2018%2E+%E2%80%94+%5BarXiv%3A1803%2E11361%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1803%2E11361%29)
44. Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurIPS, 2000. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+David+A+McAllester%2C+Satinder+P+Singh%2C+and+Yishay+Mansour%2E+Policy+gradient+methods+for+reinforcement+learning+with+function+approximation%2E+In+NeurIPS%2C+2000%2E)
45. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. [Order-embeddings of images and language](https://md.research.elamur.ai/free/en/papers/vendrov-2015-order-embeddings-images-language/). In ICLR, 2016.
46. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. MLJ, 8(3-4):229–256, 1992. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+MLJ%2C+8%283%2D4%29%3A229%E2%80%93256%2C+1992%2E)
47. Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Jiajun+Wu%2C+Joshua+B+Tenenbaum%2C+and+Pushmeet+Kohli%2E+Neural+scene+de%2Drendering%2E+In+CVPR%2C+2017%2E)
48. Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016. [find](https://scholar.google.com/scholar?q=Huijuan+Xu+and+Kate+Saenko%2E+Ask%2C+attend+and+answer%3A+Exploring+question%2Dguided+spatial+attention+for+visual+question+answering%2E+In+ECCV%2C+2016%2E)
49. Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. In NeurIPS, 2015. [find](https://scholar.google.com/scholar?q=Jimei+Yang%2C+Scott+E+Reed%2C+Ming%2DHsuan+Yang%2C+and+Honglak+Lee%2E+Weakly%2Dsupervised+disentangling+with+recurrent+transformations+for+3d+view+synthesis%2E+In+NeurIPS%2C+2015%2E)
50. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. [Stacked attention networks for image question answering](https://md.research.elamur.ai/free/en/papers/yang-2015-stacked-attention-networks-image-question/). In CVPR, 2016.
51. Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. [Neural-Symbolic VQA: Disentangling reasoning from vision and language understanding](https://md.research.elamur.ai/free/en/papers/yi-2018-neural-symbolic-vqa-disentangling-reasoning/). In NeurIPS, 2018.
52. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In CVPR, 2015.

*Updated 2 November 2026*