---
title: "CoDraw — paper by Kim et al. (2019) — summary and ideas"
description: "The authors also propose a scene similarity metric that scores both final and partial reconstructions… 58 references, cited by 41 papers here."
authors: ["Jin-Hwa Kim", "Nikita Kitaev", "Xinlei Chen", "Marcus Rohrbach", "Byoung-Tak Zhang", "Yuandong Tian", "Dhruv Batra", "Devi Parikh"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1651/"
doi: "10.18653/v1/p19-1651"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kim-2019-codraw-collaborative-drawing-as-testbed/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kim-2019-codraw-collaborative-drawing-as-testbed/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

# CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

**Authors:** Jin-Hwa Kim, [Nikita Kitaev](https://md.research.elamur.ai/free/en/authors/nikita-kitaev/), [Xinlei Chen](https://md.research.elamur.ai/free/en/authors/xinlei-chen/), [Marcus Rohrbach](https://md.research.elamur.ai/free/en/authors/marcus-rohrbach/), Byoung-Tak Zhang, [Yuandong Tian](https://md.research.elamur.ai/free/en/authors/yuandong-tian/), [Dhruv Batra](https://md.research.elamur.ai/free/en/authors/dhruv-batra/) and [Devi Parikh](https://md.research.elamur.ai/free/en/authors/devi-parikh/) · **Year:** 2019 · **Venue:** [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1651/) · [PDF](https://aclanthology.org/P19-1651.pdf) · [DOI 10.18653/v1/p19-1651](https://doi.org/10.18653/v1/p19-1651) · [OpenAlex](https://openalex.org/W2953064936)

**Citations:** 53 citations (OpenAlex, 2026-10-14) · [41](https://md.research.elamur.ai/free/en/papers/kim-2019-codraw-collaborative-drawing-as-testbed/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
CoDraw is a testbed for agents that must communicate in natural language toward a shared goal, grounded in both perception and action. In this two-player game, a Teller sees a clip art scene from the Abstract Scenes dataset and describes it through chat. A Drawer, who never sees the target, recreates it on an empty canvas by positioning, flipping and resizing clip art pieces. Human pairs played the game, producing about 10K dialogs. The authors also propose a scene similarity metric that scores both final and partial reconstructions automatically. They add a "crosstalk" protocol: Teller and Drawer models are trained on disjoint halves of the training data and evaluated together, so they cannot succeed by inventing a private code. Neural Drawer and scene2seq Teller models outperform rule-based nearest-neighbor baselines, particularly when the Teller is given an auxiliary loss and reinforcement learning fine-tuning. All models remain well below human performance. Automated crosstalk evaluation ranks Teller models in the same order as evaluation with human partners, which supports CoDraw as a practical benchmark for grounded dialog.

## Key points
- **CoDraw dataset:** 9,993 sessions (~138K messages) with the drawing state recorded at each step. Scenes come from Abstract Scenes (10,020 scenes, 58 clip arts, including Mike and Jenny in 7 poses and 5 expressions). Messages are limited to 140 characters. The data and models are released at github.com/facebookresearch/CoDraw and codraw-models.
- **Crosstalk split:** 40% Teller training (3,994), 40% Drawer training (3,995), 10% development (1,002) and 10% test (1,002).
- **Drawer results with scripted Teller messages:** the neural Drawer scores 3.39, the nearest-neighbor baseline 0.94 and live human pairs 4.17. Human Drawers given scripts reach 3.83, about 91% of the live-play score.
- **Teller results with human Drawers:** scene2seq trained by imitation alone scores 2.69, below the baseline's 3.21. Adding an auxiliary loss and RL fine-tuning raises it to 3.65, still short of human Tellers at 4.17.
- **Typical errors:**
  - The baseline Teller mentions objects that are not in the scene.
  - Imitation-only scene2seq repeats or omits objects.
  - The neural Drawer arranges pieces in semantically implausible ways, such as a hat floating in the air.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 183 works · 228,558 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 175 works · 227,440 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 133 works · 24,497 citations
- [Seoul National University](https://md.research.elamur.ai/free/en/universities/seoul-national-university/) — University, Seoul, South Korea · 41 works · 45,671 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 133 works · [⚙️ Control and Systems Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/control-and-systems-engineering/) · 45 works

## Cite

Cite as: Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra and Devi Parikh. 2019. CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kim2019codraw,
  title = {{CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication}},
  author = {Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra and Devi Parikh},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1651},
  url = {https://doi.org/10.18653/v1/p19-1651}
}
```

- [← previous A Unified Linear-Time Framework for Sentence-Level Discourse Parsing](https://md.research.elamur.ai/free/en/papers/lin-2019-unified-linear-time-framework-sentence/)
- [next → Graph-based Dependency Parsing with Graph Neural Networks](https://md.research.elamur.ai/free/en/papers/ji-2019-graph-based-dependency-parsing-graph/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

58 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018. [Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments](https://md.research.elamur.ai/free/en/papers/anderson-2017-vision-language-navigation-interpreting-visually/). In Proceedings of the IEEE Conference on Computer Vis…
2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In IEEE International Conference on Computer Vision. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C%2E+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+VQA%3A+Visual+Question+Answering%2E+In+IEEE+International+Conference+on+Computer+)
3. Lawrence W Barsalou. 1999. Perceptions of perceptual symbols. Behavioral and brain sciences, 22(4):637–660. [find](https://scholar.google.com/scholar?q=Lawrence+W+Barsalou%2E+1999%2E+Perceptions+of+perceptual+symbols%2E+Behavioral+and+brain+sciences%2C+22%284%29%3A637%E2%80%93660%2E)
4. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python. ”O’Reilly Media, Inc.”. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+2009%2E+Natural+language+processing+with+Python%2E+%E2%80%9DO%E2%80%99Reilly+Media%2C+Inc%2E%E2%80%9D%2E)
5. Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning End-to-End Goal-Oriented Dialog. In 5th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Y%2DLan+Boureau%2C+and+Jason+Weston%2E+2017%2E+Learning+End%2Dto%2DEnd+Goal%2DOriented+Dialog%2E+In+5th+International+Conference+on+Learning+Representations%2E)
6. Xinlei Chen and C Lawrence Zitnick. 2015. Mind’s eye: A recurrent visual representation for image caption generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2422–2431. [find](https://scholar.google.com/scholar?q=Xinlei+Chen+and+C+Lawrence+Zitnick%2E+2015%2E+Mind%E2%80%99s+eye%3A+A+recurrent+visual+representation+for+image+caption+generation%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+)
7. Andrea F. Daniele, Mohit Bansal, and Matthew R. Walter. 2016. Navigational instruction generation as inverse reinforcement learning with neural machine translation. CoRR, abs/1610.03164. [find](https://scholar.google.com/scholar?q=Andrea+F%2E+Daniele%2C+Mohit+Bansal%2C+and+Matthew+R%2E+Walter%2E+2016%2E+Navigational+instruction+generation+as+inverse+reinforcement+learning+with+neural+machine+translation%2E+CoRR%2C+abs%2F1610%2E03164%2E)
8. Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Samyak+Datta%2C+Georgia+Gkioxari%2C+Stefan+Lee%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+2018%2E+Embodied+question+answering%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognit)
9. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. 2017a. Visual Dialog. In IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Khushi+Gupta%2C+Avi+Singh%2C+Deshraj+Yadav%2C+Jose%CC%81+M%2E+F%2E+Moura%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+2017a%2E+Visual+Dialog%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognitio)
10. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). arXiv preprint arXiv:1612.00837.
11. Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335– 346. [find](https://scholar.google.com/scholar?q=Stevan+Harnad%2E+1990%2E+The+symbol+grounding+problem%2E+Physica+D%3A+Nonlinear+Phenomena%2C+42%281%2D3%29%3A335%E2%80%93+346%2E)
12. He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1766–1776. Association f… [find](https://scholar.google.com/scholar?q=He+He%2C+Anusha+Balakrishnan%2C+Mihail+Eric%2C+and+Percy+Liang%2E+2017%2E+Learning+symmetric+collaborative+dialogue+agents+with+dynamic+knowledge+graph+embeddings%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+t)
13. Abhishek Das, Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. 2017b. Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning. arXiv preprint arXiv:1703.06585. — [arXiv:1703.06585](https://arxiv.org/abs/1703.06585) [find](https://scholar.google.com/scholar?q=Abhishek+Das%2C+Satwik+Kottur%2C+Jose%CC%81+M%2E+F%2E+Moura%2C+Stefan+Lee%2C+and+Dhruv+Batra%2E+2017b%2E+Learning+Cooperative+Visual+Dialog+Agents+with+Deep+Reinforcement+Learning%2E+arXiv+preprint+arXiv%3A1703%2E06585%2E+%E2%80%94+%5BarXi)
14. Andrej Karpathy and Li Fei-Fei. 2015. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137.
15. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, and Jason Weston. 2016. Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems. In 4th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Andreea+Gane%2C+Xiang+Zhang%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+Alexander+Miller%2C+Arthur+Szlam%2C+and+Jason+Weston%2E+2016%2E+Evaluating+Prerequisite+Qualities+for+Learning+End%2Dto%2DEnd+Dialog+Systems%2E+)
16. Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge ‘naturally’ in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962–2967. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Satwik+Kottur%2C+Jose%CC%81+Moura%2C+Stefan+Lee%2C+and+Dhruv+Batra%2E+2017%2E+Natural+language+does+not+emerge+%E2%80%98naturally%E2%80%99+in+multi%2Dagent+dialog%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural)
17. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. [Long-term recurrent convolutional networks for visual recognition and description](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/). In Proceedings of the IEEE conference on computer vision and pattern recognition…
18. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. 2017. [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/). International Journ…
19. Jakob Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. 2016. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems 29, pages 2137–2145. [find](https://scholar.google.com/scholar?q=Jakob+Foerster%2C+Yannis+M+Assael%2C+Nando+de+Freitas%2C+and+Shimon+Whiteson%2E+2016%2E+Learning+to+Communicate+with+Deep+Multi%2DAgent+Reinforcement+Learning%2E+In+Advances+in+Neural+Information+Processing+Systems)
20. Oliver Lemon, Kallirroi Georgila, James Henderson, and Matthew Stuttle. 2006. An isu dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the talk in-car system. In Proceedings of the Eleventh Conference of the European Chapter of the Association for Comput… [find](https://scholar.google.com/scholar?q=Oliver+Lemon%2C+Kallirroi+Georgila%2C+James+Henderson%2C+and+Matthew+Stuttle%2E+2006%2E+An+isu+dialogue+system+exhibiting+reinforcement+learning+of+dialogue+policies%3A+generic+slot%2Dfilling+in+the+talk+in%2Dcar+sys)
21. Daniel Fried, Jacob Andreas, and Dan Klein. 2018a. [Unified pragmatic models for generating and following instructions](https://md.research.elamur.ai/free/en/papers/fried-2018-unified-pragmatic-models-generating-following/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1951…
22. Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018b. Speaker-follower models for vision-and-language navigation. In Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Ronghang+Hu%2C+Volkan+Cirik%2C+Anna+Rohrbach%2C+Jacob+Andreas%2C+Louis%2DPhilippe+Morency%2C+Taylor+Berg%2DKirkpatrick%2C+Kate+Saenko%2C+Dan+Klein%2C+and+Trevor+Darrell%2E+2018b%2E+Speaker%2Dfollower+models+for+v)
23. Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering. In Advances in neural information processing systems 28, pages 2296–2304. [find](https://scholar.google.com/scholar?q=Haoyuan+Gao%2C+Junhua+Mao%2C+Jie+Zhou%2C+Zhiheng+Huang%2C+Lei+Wang%2C+and+Wei+Xu%2E+2015%2E+Are+You+Talking+to+a+Machine%3F+Dataset+and+Methods+for+Multilingual+Image+Question+Answering%2E+In+Advances+in+neural+informa)
24. David Lewis. 1969. Convention: A Philosophical Study. Harvard University Press. [find](https://scholar.google.com/scholar?q=David+Lewis%2E+1969%2E+Convention%3A+A+Philosophical+Study%2E+Harvard+University+Press%2E)
25. David Lewis. 1975. Languages and language. In Keith Gunderson, editor, Minnesota Studies in the Philosophy of Science, pages 3–35. University of Minnesota Press. [find](https://scholar.google.com/scholar?q=David+Lewis%2E+1975%2E+Languages+and+language%2E+In+Keith+Gunderson%2C+editor%2C+Minnesota+Studies+in+the+Philosophy+of+Science%2C+pages+3%E2%80%9335%2E+University+of+Minnesota+Press%2E)
26. Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. [Deal or no deal? end-to-end learning of negotiation dialogues](https://md.research.elamur.ai/free/en/papers/lewis-2017-deal-or-no-deal-end/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453. Association for Computational Linguistics.
27. Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. 2017a. Dialogue Learning with Human-in-the-Loop. In 5th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Alexander+H%2E+Miller%2C+Sumit+Chopra%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Jason+Weston%2E+2017a%2E+Dialogue+Learning+with+Human%2Din%2Dthe%2DLoop%2E+In+5th+International+Conference+on+Learning+Representations%2E)
28. Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. 2017b. Learning through Dialogue Interactions by Asking Questions. In 5th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Alexander+H%2E+Miller%2C+Sumit+Chopra%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Jason+Weston%2E+2017b%2E+Learning+through+Dialogue+Interactions+by+Asking+Questions%2E+In+5th+International+Conference+on+Learning+Repr)
29. Jiwei Li, Will Monroe, Alan Ritter, and Dan Jurafsky. 2016. [Deep Reinforcement Learning for Dialogue Generation](https://md.research.elamur.ai/free/en/papers/li-2016-deep-reinforcement-learning-dialogue-generation/). In 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192– 1202.
30. Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1456– 1465. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Reginald+Long%2C+Panupong+Pasupat%2C+and+Percy+Liang%2E+2016%2E+Simpler+context%2Ddependent+logical+forms+via+model+projections%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Li)
31. Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning. In IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Caiming+Xiong%2C+Devi+Parikh%2C+and+Richard+Socher%2E+2017%2E+Knowing+When+to+Look%3A+Adaptive+Attention+via+A+Visual+Sentinel+for+Image+Captioning%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+)
32. Mateusz Malinowski and Mario Fritz. 2014. A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input. In Advances in Neural Information Processing Systems 27, pages 1682–1690. [find](https://scholar.google.com/scholar?q=Mateusz+Malinowski+and+Mario+Fritz%2E+2014%2E+A+Multi%2DWorld+Approach+to+Question+Answering+about+Real%2DWorld+Scenes+based+on+Uncertain+Input%2E+In+Advances+in+Neural+Information+Processing+Systems+27%2C+pages+)
33. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2015. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. CoRR, abs/1506.04089. [find](https://scholar.google.com/scholar?q=Hongyuan+Mei%2C+Mohit+Bansal%2C+and+Matthew+R%2E+Walter%2E+2015%2E+Listen%2C+attend%2C+and+walk%3A+Neural+mapping+of+navigational+instructions+to+action+sequences%2E+CoRR%2C+abs%2F1506%2E04089%2E)
34. Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios P. Spithourakis, and Lucy Vanderwende. 2017. [Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation](https://md.research.elamur.ai/free/en/papers/mostafazadeh-2017-image-grounded-conversations-multimodal-context/). arXiv preprint arXiv:1701.08251.
35. Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Exploring Models and Data for Image Question Answering. In Advances in Neural Information Processing Systems 28, pages 2935–2943. [find](https://scholar.google.com/scholar?q=Mengye+Ren%2C+Ryan+Kiros%2C+and+Richard+Zemel%2E+2015%2E+Exploring+Models+and+Data+for+Image+Question+Answering%2E+In+Advances+in+Neural+Information+Processing+Systems+28%2C+pages+2935%E2%80%932943%2E)
36. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. [Neural Responding Machine for Short-Text Conversation](https://md.research.elamur.ai/free/en/papers/shang-2015-neural-responding-machine-short-text/). In 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1577–1586.
37. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and William B. Dolan. 2015. [A Neural Network Approach to Context-Sensitive Generation of Conversational Responses](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/). In 2015 Annual Conference of the North American Chapter of t…
38. Florian Strub, Harm de Vries, Jeremie Mary, Bilal Piot, Aaron Courville, and Olivier Pietquin. 2017. End-to-end optimization of goal-driven and visually grounded dialogue systems. arXiv preprint arXiv:1703.05423. — [arXiv:1703.05423](https://arxiv.org/abs/1703.05423) [find](https://scholar.google.com/scholar?q=Florian+Strub%2C+Harm+de+Vries%2C+Jeremie+Mary%2C+Bilal+Piot%2C+Aaron+Courville%2C+and+Olivier+Pietquin%2E+2017%2E+End%2Dto%2Dend+optimization+of+goal%2Ddriven+and+visually+grounded+dialogue+systems%2E+arXiv+preprint+arXiv)
39. Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. 2016. Learning Multiagent Communication with Backpropagation. In Advances in Neural Information Processing SystemsNeural Information Processing Systems 29, pages 2244–2252. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Arthur+Szlam%2C+and+Rob+Fergus%2E+2016%2E+Learning+Multiagent+Communication+with+Backpropagation%2E+In+Advances+in+Neural+Information+Processing+SystemsNeural+Information+Processing+Syst)
40. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. MovieQA: Understanding Stories in Movies through Question-Answering. In IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Makarand+Tapaswi%2C+Yukun+Zhu%2C+Rainer+Stiefelhagen%2C+Antonio+Torralba%2C+Raquel+Urtasun%2C+and+Sanja+Fidler%2E+2016%2E+MovieQA%3A+Understanding+Stories+in+Movies+through+Question%2DAnswering%2E+In+IEEE+Conference+on+C)
41. Stefanie A Tellex, Thomas Fleming Kollar, Steven R Dickerson, Matthew R Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Twenty-Fifth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Stefanie+A+Tellex%2C+Thomas+Fleming+Kollar%2C+Steven+R+Dickerson%2C+Matthew+R+Walter%2C+Ashis+Banerjee%2C+Seth+Teller%2C+and+Nicholas+Roy%2E+2011%2E+Understanding+natural+language+commands+for+robotic+navigation+and+)
42. Orioi Vinyals and Quoc V. Le. 2015. [find](https://scholar.google.com/scholar?q=Orioi+Vinyals+and+Quoc+V%2E+Le%2E+2015%2E)
43. [A Neural Conversational Model](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/). In ICML Deep Learning Workshop 2015.
44. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2017. Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge. IEEE transactions on pattern analysis and machine intelligence, 39(4):652–663. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Alexander+Toshev%2C+Samy+Bengio%2C+and+Dumitru+Erhan%2E+2017%2E+Show+and+Tell%3A+Lessons+learned+from+the+2015+MSCOCO+Image+Captioning+Challenge%2E+IEEE+transactions+on+pattern+analysis+and+machine)
45. Adam Vogel and Daniel Jurafsky. 2010. Learning to follow navigational directions. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 806–814. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Vogel+and+Daniel+Jurafsky%2E+2010%2E+Learning+to+follow+navigational+directions%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+806%E2%80%93814%2E+Association)
46. Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2016. GuessWhat?! Visual object discovery through multi-modal dialogue. arXiv preprint arXiv:1611.08481. [find](https://scholar.google.com/scholar?q=Harm+de+Vries%2C+Florian+Strub%2C+Sarath+Chandar%2C+Olivier+Pietquin%2C+Hugo+Larochelle%2C+and+Aaron+Courville%2E+2016%2E+GuessWhat%3F%21+Visual+object+discovery+through+multi%2Dmodal+dialogue%2E+arXiv+preprint+arXiv%3A1611%2E)
47. Matthew R. Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller. 2014. A framework for learning semantic maps from grounded natural language descriptions. The International Journal of Robotics Research, 33(9):1167– 1190. [find](https://scholar.google.com/scholar?q=Matthew+R%2E+Walter%2C+Sachithra+Hemachandra%2C+Bianca+Homberg%2C+Stefanie+Tellex%2C+and+Seth+Teller%2E+2014%2E+A+framework+for+learning+semantic+maps+from+grounded+natural+language+descriptions%2E+The+International+)
48. Sida I. Wang, Samuel Ginn, Percy Liang, and Christoper D. Manning. 2017. Naturalizing a Programming Language via Interactive Learning. In 55th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sida+I%2E+Wang%2C+Samuel+Ginn%2C+Percy+Liang%2C+and+Christoper+D%2E+Manning%2E+2017%2E+Naturalizing+a+Programming+Language+via+Interactive+Learning%2E+In+55th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
49. Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning Language Games through Interaction. In 54th Annual Meeting of the Association for Computational Linguistics, pages 2368–2378. [find](https://scholar.google.com/scholar?q=Sida+I%2E+Wang%2C+Percy+Liang%2C+and+Christopher+D%2E+Manning%2E+2016%2E+Learning+Language+Games+through+Interaction%2E+In+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+2368%E2%80%932378%2E)
50. Zhuoran Wang and Oliver Lemon. 2013. A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the believability of observed information. In SIGDIAL 2013 Conference, pages 423–432. [find](https://scholar.google.com/scholar?q=Zhuoran+Wang+and+Oliver+Lemon%2E+2013%2E+A+simple+and+generic+belief+tracking+mechanism+for+the+dialog+state+tracking+challenge%3A+On+the+believability+of+observed+information%2E+In+SIGDIAL+2013+Conference%2C+p)
51. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. [find](https://scholar.google.com/scholar?q=Ronald+J+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+learning%2C+8%283%2D4%29%3A229%E2%80%93256%2E)
52. Kelvin Xu, Aaron Courville, Richard S Zemel, and Yoshua Bengio. 2015. [Show, Attend and Tell : Neural Image Caption Generation with Visual Attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). In 32nd International Conference on Machine Learning.
53. Licheng Yu, Eunbyung Park, Alexander C Berg, and Tamara L. Berg. 2015. Visual Madlibs : Fill in the blank Description Generation and Question Answering. In IEEE International Conference on Computer Vision, pages 2461–2469. [find](https://scholar.google.com/scholar?q=Licheng+Yu%2C+Eunbyung+Park%2C+Alexander+C+Berg%2C+and+Tamara+L%2E+Berg%2E+2015%2E+Visual+Madlibs+%3A+Fill+in+the+blank+Description+Generation+and+Question+Answering%2E+In+IEEE+International+Conference+on+Computer+Vi)
54. Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and Yang: Balancing and Answering Binary Visual Questions. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5014–5022. [find](https://scholar.google.com/scholar?q=Peng+Zhang%2C+Yash+Goyal%2C+Douglas+Summers%2DStay%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+2016%2E+Yin+and+Yang%3A+Balancing+and+Answering+Binary+Visual+Questions%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recog)
55. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. [Visual7W: Grounded Question Answering in Images](https://md.research.elamur.ai/free/en/papers/zhu-2015-visual7w-grounded-question-answering-images/). In IEEE Conference on Computer Vision and Pattern Recognition, pages 4995–5004.
56. C. Lawrence Zitnick and Devi Parikh. 2013. Bringing semantics into focus using visual abstraction. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3009–3016. [find](https://scholar.google.com/scholar?q=C%2E+Lawrence+Zitnick+and+Devi+Parikh%2E+2013%2E+Bringing+semantics+into+focus+using+visual+abstraction%2E+Proceedings+of+the+IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages)
57. C. Lawrence Zitnick, Devi Parikh, and Lucy Vanderwende. 2013. Learning the visual interpretation of sentences. Proceedings of the IEEE International Conference on Computer Vision, pages 1681–1688. [find](https://scholar.google.com/scholar?q=C%2E+Lawrence+Zitnick%2C+Devi+Parikh%2C+and+Lucy+Vanderwende%2E+2013%2E+Learning+the+visual+interpretation+of+sentences%2E+Proceedings+of+the+IEEE+International+Conference+on+Computer+Vision%2C+pages+1681%E2%80%931688%2E)
58. Chat to Complete!  Instructions  Instructions for Teller \#\# 1 Your fellow Turker will ask you questions about your secret scene. 2 Your objective is to help the fellow Turker recreate the scene. You typically describe the details of the image and/or answer their questions. [find](https://scholar.google.com/scholar?q=Chat+to+Complete%21+%05+Instructions+%05+Instructions+for+Teller+%5C%23%5C%23+1+Your+fellow+Turker+will+ask+you+questions+about+your+secret+scene%2E+2+Your+objective+is+to+help+the+fellow+Turker+recreate+the+scene%2E+Y)

*Updated 29 October 2026*