---
title: "Pragmatic Issue-Sensitive Image Captioning — Nie 2020"
description: "It introduces Issue-Sensitive Image Captioning (ISIC). 34 references, cited by 10 papers here."
authors: ["Allen Nie", "Reuben Cohn-Gordon", "Christopher Potts"]
year: 2020
venue: "Findings of the Association for Computational Linguistics: EMNLP 2020"
source: "https://aclanthology.org/2020.findings-emnlp.173/"
doi: "10.18653/v1/2020.findings-emnlp.173"
arxiv: "2004.14451"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nie-2020-pragmatic-issue-sensitive-image-captioning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nie-2020-pragmatic-issue-sensitive-image-captioning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/) / Pragmatic Issue-Sensitive Image Captioning

# Pragmatic Issue-Sensitive Image Captioning

**Authors:** [Allen Nie](https://md.research.elamur.ai/free/en/authors/allen-nie/), [Reuben Cohn-Gordon](https://md.research.elamur.ai/free/en/authors/reuben-cohn-gordon/) and [Christopher Potts](https://md.research.elamur.ai/free/en/authors/christopher-potts/) · **Year:** 2020 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2020](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2020](https://md.research.elamur.ai/free/en/venues/findings/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.findings-emnlp.173/) · [PDF](https://aclanthology.org/2020.findings-emnlp.173.pdf) · [DOI 10.18653/v1/2020.findings-emnlp.173](https://doi.org/10.18653/v1/2020.findings-emnlp.173) · [arXiv 2004.14451](https://arxiv.org/abs/2004.14451) · [OpenAlex](https://openalex.org/W3099883243)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/nie-2020-pragmatic-issue-sensitive-image-captioning/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper argues that a good caption must be both true and relevant to what the reader currently cares about. It introduces Issue-Sensitive Image Captioning (ISIC). The input is a target image plus an "issue": a set of images split into equivalence classes. The caption should say which class the target falls into, and say it accurately. The authors need no new training data for this. They take pretrained neural captioners, treat them as the base speaker in the Rational Speech Acts (RSA) framework, and add the partition when decoding. A pragmatic listener built with Bayes' rule judges how well a caption separates the target's cell from the other cells.

The main experiments use CUB, whose rich attribute labels make issues easy to build and check automatically. A Mechanical Turk study found the issue-aware models clearly more issue-sensitive than plain captioners. The method was also run on MS COCO, with issues built from VQA 2.0 questions. This links captioning to VQA and points toward controllable generation more broadly.

## Key points
- Issues are modelled as partitions, following linguistic work on Questions Under Discussion, with Lewis (1988) as the source for the partition idea. Each cell stands for one possible resolution of the issue.
- For the base speakers S0, the authors use captioning models from Hendricks et al. (2016) and from Rennie et al. (2017). On top of them sit a pragmatic RSA speaker S1 and issue-sensitive variants S1^C and S1^C+H.
- Human study design:
  - 105 Mechanical Turk workers gave 1,365 responses.
  - Workers saw only a question and a caption, never the image.
  - By Fisher's exact test, the two issue-sensitive models differed significantly (p \< 0.05) from S0, S1 and human captions.
- The automatic feature-in-text classifier agreed with human judgments 90.1–94.9% of the time, with 273 items per caption source. Agreement was similar for S0 and S1^C+H.
- MS COCO / VQA 2.0 setup:
  - Partitions were built by gathering all images that share a VQA 2.0 question (exact string match) and grouping them by their ground-truth answer.
  - The base model was a 6-layer Transformer trained with self-critical sequence training, scoring CIDEr 1.29, decoded with beam size 5.
  - Code is released at github.com/windweller/Pragmatic-ISIC.

## Interesting ideas
- An issue is a partition over alternatives, and an utterance counts as relevant when it pins down the cell holding the target; this makes relevance a precise, checkable notion.
- Pragmatic reasoning can be added only at decoding time on a frozen pretrained speaker, by scoring candidates with a Bayesian listener that compares the target against distractor cells.
- Issues can be built cheaply from existing QA annotations: collect items that share the same question and split them by answer.
- Before trusting an automatic judge, compare it with human decisions separately for each system, so it does not favour one model over another.
- How issue-sensitive a model can become is proposed as a test of how well it breaks content into objects and abstract concepts.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): crate prag could represent the Question Under Discussion as a partition over candidate world states. Implicature and relevance decisions would then reduce to asking which cell an utterance selects, which keeps them explainable.
- **I2** (back): the generator could rerank its candidate sentences by how well a literal reader picks out the intended entity or state from alternative world states, with no retraining.
- **I3** (answer): fable QA or school-test items could be turned into contrast sets in the same way. These would test whether a reader's answer truly separates the right world state from its look-alikes.
- **I4** (learn): silver-data checkers and automatic evaluators could get a per-system agreement audit against a small human-labelled sample, alongside negative controls.
- **I5** (world): partition-based probes could test whether the world-of-text builder keeps entities, attributes and states separate enough to tell close alternatives apart.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 269 works

## Cite

Cite as: Allen Nie, Reuben Cohn-Gordon and Christopher Potts. 2020. Pragmatic Issue-Sensitive Image Captioning. Findings of the Association for Computational Linguistics: EMNLP 2020.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nie2020pragmatic,
  title = {{Pragmatic Issue-Sensitive Image Captioning}},
  author = {Allen Nie, Reuben Cohn-Gordon and Christopher Potts},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.173},
  eprint = {2004.14451},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.173}
}
```

- [← previous On the Frailty of Universal POS Tags for Neural UD Parsers](https://md.research.elamur.ai/free/en/papers/anderson-2020-frailty-universal-pos-tags-neural/)
- [next → pyBART: Evidence-based Syntactic Transformations for IE](https://md.research.elamur.ai/free/en/papers/tiktinsky-2020-pybart-evidence-based-syntactic-transformations/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/)

## References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Xiaodong+He%2C+Chris+Buehler%2C+Damien+Teney%2C+Mark+Johnson%2C+Stephen+Gould%2C+and+Lei+Zhang%2E+2018%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+visual+question+answering%2E+In+Proc)
2. Jacob Andreas and Dan Klein. 2016. [Reasoning about pragmatics with neural listeners and speakers](https://md.research.elamur.ai/free/en/papers/andreas-2016-reasoning-about-pragmatics-neural-listeners/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Austin, Texas. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d16-1125)
3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425–2433. [↗](https://doi.org/10.1007/s11263-016-0966-6)
4. Reuben Cohn-Gordon and Noah Goodman. 2019. Lost in machine translation: A method to reduce meaning loss. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 437–441, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1042)
5. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. [Pragmatically informative image captioning with character-level inference](https://md.research.elamur.ai/free/en/papers/gordon-2018-pragmatically-informative-image-captioning-character/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 439–443, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2070)
6. Michael C. Frank and Noah D. Goodman. 2012. [Predicting pragmatic reasoning in language games](https://md.research.elamur.ai/free/en/papers/frank-2012-predicting-pragmatic-reasoning-language-games/). Science, 336(6084):998. [↗](https://doi.org/10.1126/science.1218633)
7. Michael Franke. 2009. Signal to Act: Game Theory in Pragmatics. ILLC Dissertation Series. Institute for Logic, Language and Computation, University of Amsterdam. [find](https://scholar.google.com/scholar?q=Michael+Franke%2E+2009%2E+Signal+to+Act%3A+Game+Theory+in+Pragmatics%2E+ILLC+Dissertation+Series%2E+Institute+for+Logic%2C+Language+and+Computation%2C+University+of+Amsterdam%2E)
8. Daniel Fried, Jacob Andreas, and Dan Klein. 2018. [Unified pragmatic models for generating and following instructions](https://md.research.elamur.ai/free/en/papers/fried-2018-unified-pragmatic-models-generating-following/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1951–1963, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1711.04987)
9. Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. 2017. Stylenet: Generating attractive visual captions with styles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3137–3146. [find](https://scholar.google.com/scholar?q=Chuang+Gan%2C+Zhe+Gan%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+and+Li+Deng%2E+2017%2E+Stylenet%3A+Generating+attractive+visual+captions+with+styles%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Rec)
10. Jonathan Ginzburg. 1996. Dynamics and the semantics of dialogue. In Jerry Seligman, editor, Language, Logic, and Computation, volume 1, pages 221–237. CSLI, Stanford, CA. [find](https://scholar.google.com/scholar?q=Jonathan+Ginzburg%2E+1996%2E+Dynamics+and+the+semantics+of+dialogue%2E+In+Jerry+Seligman%2C+editor%2C+Language%2C+Logic%2C+and+Computation%2C+volume+1%2C+pages+221%E2%80%93237%2E+CSLI%2C+Stanford%2C+CA%2E)
11. Dave Golland, Percy Liang, and Dan Klein. 2010. A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 410–419, Cambridge, MA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dave+Golland%2C+Percy+Liang%2C+and+Dan+Klein%2E+2010%2E+A+game%2Dtheoretic+approach+to+generating+spatial+descriptions%2E+In+Proceedings+of+the+2010+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C)
12. Noah D. Goodman and Michael C. Frank. 2016. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20(11):818– 829. [↗](https://doi.org/10.1016/j.tics.2016.08.005)
13. Noah D. Goodman and Daniel Lassiter. 2015. Probabilistic semantics and pragmatics. In Shalom Lappin and Chris Fox, editors, The Handbook of Contemporary Semantic Theory, 2 edition. Wiley-Blackwell, Oxford. [find](https://scholar.google.com/scholar?q=Noah+D%2E+Goodman+and+Daniel+Lassiter%2E+2015%2E+Probabilistic+semantics+and+pragmatics%2E+In+Shalom+Lappin+and+Chris+Fox%2C+editors%2C+The+Handbook+of+Contemporary+Semantic+Theory%2C+2+edition%2E+Wiley%2DBlackwell%2C+Ox)
14. Noah D. Goodman and Andreas Stuhlmüller. 2013. Knowledge and implicature: Modeling language understanding as social cognition. Topics in Cognitive Science, 5(1):173–184. [↗](https://doi.org/10.1111/tops.12007)
15. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913. [↗](https://doi.org/10.1007/s11263-018-1116-0)
16. H. Paul Grice. 1975. Logic and conversation. In Peter Cole and Jerry Morgan, editors, Syntax and Semantics, volume 3: Speech Acts, pages 43–58. Academic Press, New York. [↗](https://doi.org/10.1163/9789004368811_003)
17. Jeroen Groenendijk and Martin Stokhof. 1984. Studies in the Semantics of Questions and the Pragmatics of Answers. Ph.D. thesis, University of Amsterdam. [find](https://scholar.google.com/scholar?q=Jeroen+Groenendijk+and+Martin+Stokhof%2E+1984%2E+Studies+in+the+Semantics+of+Questions+and+the+Pragmatics+of+Answers%2E+Ph%2ED%2E+thesis%2C+University+of+Amsterdam%2E)
18. Robert X. D. Hawkins and Noah D. Goodman. 2019. Why do you ask? The informational dynamics of questions and answers. PsyArXiv, 6. [find](https://scholar.google.com/scholar?q=Robert+X%2E+D%2E+Hawkins+and+Noah+D%2E+Goodman%2E+2019%2E+Why+do+you+ask%3F+The+informational+dynamics+of+questions+and+answers%2E+PsyArXiv%2C+6%2E)
19. Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In European Conference on Computer Vision, pages 3–19. Springer. [↗](https://doi.org/10.48550/arxiv.1603.08507)
20. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
21. Gerhard Jäger. 2007. Game dynamics connects semantics and pragmatics. In Ahti-Veikko Pietarinen, editor, Game Theory and Linguistic Meaning, pages 89–102. Elsevier, Amsterdam. [find](https://scholar.google.com/scholar?q=Gerhard+Ja%CC%88ger%2E+2007%2E+Game+dynamics+connects+semantics+and+pragmatics%2E+In+Ahti%2DVeikko+Pietarinen%2C+editor%2C+Game+Theory+and+Linguistic+Meaning%2C+pages+89%E2%80%93102%2E+Elsevier%2C+Amsterdam%2E)
22. Justine T Kao, Jean Y Wu, Leon Bergen, and Noah D Goodman. 2014. [Nonliteral understanding of number words](https://md.research.elamur.ai/free/en/papers/kao-2014-nonliteral-understanding-number-words/). Proceedings of the National Academy of Sciences, 111(33):12002–12007. [↗](https://doi.org/10.1073/pnas.1407479111)
23. Andrej Karpathy and Li Fei-Fei. 2015. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3128–3137. [↗](https://doi.org/10.1109/tpami.2016.2598339)
24. David Lewis. 1969. Convention. Harvard University Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=David+Lewis%2E+1969%2E+Convention%2E+Harvard+University+Press%2C+Cambridge%2C+MA%2E)
25. David Lewis. 1988. Relevant implication. Theoria, 54(3):161–174. [find](https://scholar.google.com/scholar?q=David+Lewis%2E+1988%2E+Relevant+implication%2E+Theoria%2C+54%283%29%3A161%E2%80%93174%2E)
26. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European conference on computer vision, pages 740–755. Springer. [↗](https://doi.org/10.4230/lipics.cosit.2024.1)
27. Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. 2018. Discriminability objective for training descriptive captions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6964–6974. [find](https://scholar.google.com/scholar?q=Ruotian+Luo%2C+Brian+Price%2C+Scott+Cohen%2C+and+Gregory+Shakhnarovich%2E+2018%2E+Discriminability+objective+for+training+descriptive+captions%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Patte)
28. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. [Generation and comprehension of unambiguous object descriptions](https://md.research.elamur.ai/free/en/papers/mao-2015-generation-comprehension-unambiguous-object-descriptions/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20. IEEE.
29. Alexander Mathews, Lexing Xie, and Xuming He. 2015. Senticap: Generating image descriptions with sentiments. arXiv preprint arXiv:1510.01431. [↗](https://arxiv.org/abs/1510.01431)
30. Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. [Colors in context: A pragmatic neural model for grounded language understanding](https://md.research.elamur.ai/free/en/papers/monroe-2017-colors-context-pragmatic-neural-model/). Transactions of the Association for Computational Linguistics, 5:325–338. [↗](https://doi.org/10.1162/tacl_a_00064)
31. Will Monroe and Christopher Potts. 2015. Learning in the Rational Speech Acts model. In Proceedings of 20th Amsterdam Colloquium, Amsterdam. ILLC. [↗](https://doi.org/10.53437/ph73fp56)
32. Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 49–58. [↗](https://doi.org/10.48550/arxiv.1605.05395)
33. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. [Self-critical sequence training for image captioning](https://md.research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024.
34. Craige Roberts. 1996. Information structure: Towards an integrated formal theory of pragmatics. In Jae Hak Yoon and Andreas Kathol, editors, OSU Working Papers in Linguistics, volume 49: Papers in Semantics, pages 91–136. The Ohio State University Department of Linguistics, Columbus, OH. A [find](https://scholar.google.com/scholar?q=Craige+Roberts%2E+1996%2E+Information+structure%3A+Towards+an+integrated+formal+theory+of+pragmatics%2E+In+Jae+Hak+Yoon+and+Andreas+Kathol%2C+editors%2C+OSU+Working+Papers+in+Linguistics%2C+volume+49%3A+Papers+in+Sem)

*Updated 24 October 2026*