--- title: "Pragmatically Informative Image — Cohn-Gordon 2018" description: "Standard neural image captioners describe an image accurately but often fail to say what sets it apart from similar pictures." authors: ["Reuben Cohn-Gordon", "Noah Goodman", "Christopher Potts"] year: 2018 venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)" source: "https://aclanthology.org/N18-2070/" doi: "10.18653/v1/n18-2070" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/gordon-2018-pragmatically-informative-image-captioning-character/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gordon-2018-pragmatically-informative-image-captioning-character/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Pragmatically Informative Image Captioning with Character-Level Inference # Pragmatically Informative Image Captioning with Character-Level Inference **Authors:** [Reuben Cohn-Gordon](https://md.research.elamur.ai/free/en/authors/reuben-cohn-gordon/), [Noah Goodman](https://md.research.elamur.ai/free/en/authors/noah-goodman/), [Christopher Potts](https://md.research.elamur.ai/free/en/authors/christopher-potts/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/N18-2070/) · [PDF](https://aclanthology.org/N18-2070.pdf) · [DOI 10.18653/v1/n18-2070](https://doi.org/10.18653/v1/n18-2070) · [OpenAlex](https://openalex.org/W2797753589) **Citations:** 62 citations (openalex-snapshot · 2026-10-14) · [41](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Standard neural image captioners describe an image accurately but often fail to say what sets it apart from similar pictures. This paper adds a Rational Speech Acts (RSA) model to a captioner. RSA has the speaker reason about a listener who must pick out the target image from a set of distractors. Earlier neural RSA work had to normalize over all possible captions, which is intractable, so it relied on a small sampled set of candidate utterances. The authors instead apply RSA incrementally, one character at a time, using a character-level LSTM in a CNN-RNN captioner. At each step the speaker chooses among roughly 30 symbols, and the listener's beliefs about the images carry over from step to step. The models are trained on Visual Genome regions and MSCOCO. Evaluation is automatic: a separately trained listener model tries to identify the target among 10 similar images. The pragmatic character-level speaker identifies targets far more reliably than the literal captioner and also beats a word-level RSA variant. This shows that local character choices can produce caption-level pragmatic effects cheaply. ## Key points - Accuracy on TS1 / TS2 (each 100 clusters of 10 images): - Char S0: 48.9 / 47.5 - Char S1: 68.0 / 65.9 - Word S0: 57.6 / 53.4 - Word S1: 60.6 / 57.6 - The word S0 is stronger than the char S0, but the char S1 beats the word S1. The authors attribute this to the many more decision points where pragmatic reasoning can act. - The search space per step is about 30 characters, compared with about 20,000 words. Settings used: beam width 10 and rationality α = 5.0. - Consider the cases where S1 succeeded and S0 failed. In 66% of them, the S1 caption was not among S0's top-50 beam outputs, so a sampling-based (non-incremental) RSA could not have produced it. - Training data were split in half, one part for the caption generator and one for the evaluation listener L_eval. The code is released as Recurrent-RSA on GitHub. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cited by Cited by 41 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 30 not yet. - [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://md.research.elamur.ai/free/en/papers/fried-2023-pragmatics-language-grounding-phenomena-tasks/) - [An Incremental Iterated Response Model of Pragmatics](https://md.research.elamur.ai/free/en/papers/gordon-2019-incremental-iterated-response-model-pragmatics/) - [Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness](https://md.research.elamur.ai/free/en/papers/kim-2020-will-i-sound-like-me/) - [Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://md.research.elamur.ai/free/en/papers/kim-2021-perspective-taking-pragmatics-generating-empathetic/) - [GeDi: Generative Discriminator Guided Sequence Generation](https://md.research.elamur.ai/free/en/papers/krause-2021-gedi-generative-discriminator-guided-sequence/) - [PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations](https://md.research.elamur.ai/free/en/papers/qi-2023-pragmaticqa-dataset-pragmatic-question-answering/) - [Harnessing the linguistic signal to predict scalar inferences](https://md.research.elamur.ai/free/en/papers/schuster-2020-harnessing-linguistic-signal-predict-scalar/) - [A practical introduction to the Rational Speech Act modeling framework](https://md.research.elamur.ai/free/en/papers/scontras-2021-practical-introduction-rational-speech-act/) - [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://md.research.elamur.ai/free/en/papers/wang-2024-rsa-control-pragmatics-grounded-lightweight/) - [FUDGE: Controlled Text Generation With Future Discriminators](https://md.research.elamur.ai/free/en/papers/yang-2021-fudge-controlled-text-generation-future/) - [Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories](https://md.research.elamur.ai/free/en/papers/zarrie-2019-know-what-you-don-t/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{cohngordon2018pragmatically, title = {{Pragmatically Informative Image Captioning with Character-Level Inference}}, author = {Cohn-Gordon, Reuben and Goodman, Noah and Potts, Christopher}, booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)}, year = {2018}, doi = {10.18653/v1/n18-2070}, url = {https://doi.org/10.18653/v1/n18-2070} } ``` - [← previous Planning, Inference and Pragmatics in Sequential Language Games](https://md.research.elamur.ai/free/en/papers/khani-2018-planning-inference-pragmatics-sequential-language/) - [next → PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/chen-2018-preco-large-scale-dataset-preschool/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) ## References 10 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jacob+Andreas+and+Dan+Klein%2E+2016%2E+Reasoning+about+pragmatics+with+neural+listeners+and+speakers%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1173%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/andreas-2016-reasoning-about-pragmatics-neural-listeners/) 2. Will Monroe, Robert X. D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338. [↗](https://doi.org/10.1162/tacl_a_00064) [our card](https://md.research.elamur.ai/free/en/papers/monroe-2017-colors-context-pragmatic-neural-model/) 3. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. [↗](https://arxiv.org/abs/1504.00325) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/) 4. Will Monroe and Christopher Potts. 2015. Learning in the Rational Speech Acts model. In Proceedings of 20th Amsterdam Colloquium, Amsterdam. ILLC. [find](https://scholar.google.com/scholar?q=Will+Monroe+and+Christopher+Potts%2E+2015%2E+Learning+in+the+Rational+Speech+Acts+model%2E+In+Proceedings+of+20th+Amsterdam+Colloquium%2C+Amsterdam%2E+ILLC%2E) 5. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive Science, 19(2):233–263. [↗](https://doi.org/10.1207/s15516709cog1902_3) [our card](https://md.research.elamur.ai/free/en/papers/dale-1995-computational-interpretations-gricean-maxims-generation/) 6. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. arXiv preprint arXiv:1701.02870. [↗](https://arxiv.org/abs/1701.02870) 7. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In European Conference on Computer Vision, pages 15–29. Springer. [find](https://scholar.google.com/scholar?q=Ali+Farhadi%2C+Mohsen+Hejrati%2C+Mohammad+Amin+Sadeghi%2C+Peter+Young%2C+Cyrus+Rashtchian%2C+Julia+Hockenmaier%2C+and+David+Forsyth%2E+2010%2E+Every+picture+tells+a+story%3A+Generating+sentences+from+images%2E+In+Europea) 8. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Alexander+Toshev%2C+Samy+Bengio%2C+and+Dumitru+Erhan%2E+2015%2E+Show+and+tell%3A+A+neural+image+caption+generator%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/) 9. Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998. [find](https://scholar.google.com/scholar?q=Michael+C%2E+Frank+and+Noah+D%2E+Goodman%2E+2012%2E+Predicting+pragmatic+reasoning+in+language+games%2E+Science%2C+336%286084%29%3A998%2E) [our card](https://md.research.elamur.ai/free/en/papers/frank-2012-predicting-pragmatic-reasoning-language-games/) 10. Noah D Goodman and Andreas Stuhlmüller. 2013. Knowledge and implicature: Modeling language understanding as social cognition. Topics in Cognitive Science, 5(1):173–184. [find](https://scholar.google.com/scholar?q=Noah+D+Goodman+and+Andreas+Stuhlmu%CC%88ller%2E+2013%2E+Knowledge+and+implicature%3A+Modeling+language+understanding+as+social+cognition%2E+Topics+in+Cognitive+Science%2C+5%281%29%3A173%E2%80%93184%2E)