--- title: "Know What You Don't Know — Zarrieß 2019" description: "A speaker must refer to an object in an image whose category the speaker does not know, for example a bus when only… 23 references, cited by 9 papers here." authors: ["Sina Zarrieß", "David Schlangen"] year: 2019 venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P19-1063/" doi: "10.18653/v1/p19-1063" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zarriess-2019-know-what-you-don-t/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zarriess-2019-know-what-you-don-t/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories # Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories **Authors:** [Sina Zarrieß](https://md.research.elamur.ai/free/en/authors/sina-zarriess/) and [David Schlangen](https://md.research.elamur.ai/free/en/authors/david-schlangen/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), pages 654–659 **Link:** [aclanthology.org/P19-1063](https://aclanthology.org/P19-1063/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology) [**15** citations](https://openalex.org/works?filter=cites:W2951064641) OpenAlex, 2026-10-14 · [**9** citations in our library](https://md.research.elamur.ai/free/en/papers/zarriess-2019-know-what-you-don-t/cited-by/) of the 140,884 papers whose bibliographies we read · [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://aclanthology.org/volumes/P19-1/) · [licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/P19-1063/) - [PDF](https://aclanthology.org/P19-1063.pdf) - [DOI 10.18653/v1/p19-1063](https://doi.org/10.18653/v1/p19-1063) - [OpenAlex](https://openalex.org/W2951064641) **Authors (2)** 1. [Sina Zarrieß](https://openalex.org/A5078051602) [ORCID](https://orcid.org/0000-0002-1384-1218) · [Semantic Scholar](https://www.semanticscholar.org/author/1721364) · h 13 · 479 citations · 161 works 2. [David Schlangen](https://openalex.org/A5032801642) [ORCID](https://orcid.org/0000-0002-2686-6887) · [Semantic Scholar](https://www.semanticscholar.org/author/1817455) · h 28 · 3,044 citations · 255 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about A speaker must refer to an object in an image whose category the speaker does not know, for example a bus when only cars were seen in training. A neural literal speaker trained on known categories tends to name the wrong category. The paper adds a pragmatic decoder in the style of Rational Speech Acts: the listener keeps a belief over the object's category, and the speaker chooses each word by a score combining the literal speaker and the listener, with weights that favour words not yet used. Uncertainty is set to uniform over the categories. Six medium-frequency categories (cat, horse, cup, bottle, bus, train) are held out from the RefCOCO training data. In Table 1, the share of expressions that contain a wrong category noun falls for all six categories: for example from 39.8% to 16.6% for bottle and from 62.7% to 33.2% for cup. The share of expressions with no noun at all rises, from 27.5% to 56.2% for bottle. The reference-resolution test uses a listener that knows the true categories, to avoid training the evaluator on the same mistakes. The results are mixed across the two test sets; the authors call them positive evidence for communicative success. ## Ideas - I4. Build a zero-shot split for the school lessons: hold out one set of object words from the training lessons, and test whether the answer avoids them. Split by lesson, not by sentence, and report on the held-out lessons only. ## For Mova - I1. `prag/src/rsa.rs`: a greedy word-level pragmatic speaker with two fixed weights, one for the literal score and one for the listener score, and a penalty that grows for words already used in the sentence. Keep the score table in a `BTreeMap` and check that two runs give the same text. - I2. Hedging for unknown words: when a noun in the input is outside the dictionary (`en/src/dict.rs`), prefer a general word ("thing") plus attributes, and count how often a wrong category name is produced for such objects in the school texts. - I3. Evaluate with a listener that knows the true categories (a rule-based checker such as `coref/src/eval.rs` style scoring, not a model trained on the same text), as the paper does. This avoids the circularity the authors describe. Report resolution accuracy with the reference speaker as a negative control. The pragmatic speaker belongs with `prag/src/rsa.rs` and `prag/src/roles.rs`. Unknown-word handling relates to `en/src/dict.rs` and [languages/en/19-lexicon-mwe-register.md](https://mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/). Reference and the choice of noun belong in [languages/en/35-reference-and-coreference.md](https://mova.elamur.ai/free/en/docs/languages/en/35-reference-and-coreference/), and the evaluation idea fits `coref/src/eval.rs`. Measures: the wrong-category rate and the no-noun rate on held-out lessons, the resolution accuracy with a listener that knows the categories, run twice for determinism. ## Institutions [🎓**University of Potsdam** Potsdam · Germany 31 works · 4,698 citations](https://md.research.elamur.ai/free/en/universities/university-of-potsdam/) · [🎓**Bielefeld University** Bielefeld · Germany 12 works · 574 citations](https://md.research.elamur.ai/free/en/universities/bielefeld-university/) ## Cite Cite as Sina Zarrieß and David Schlangen. 2019. Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/p19-1063](https://doi.org/10.18653/v1/p19-1063) ```bibtex @inproceedings{zarriess2019know, title = {{Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories}}, author = {Sina Zarrieß and David Schlangen}, booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, year = {2019}, doi = {10.18653/v1/p19-1063}, url = {https://doi.org/10.18653/v1/p19-1063} } ``` - [← previous Concord and agreement features in Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/winchester-2019-concord-agreement-features-modern-standard/) - [next → The Role of Pragmatic and Discourse Context in Determining Argument Impact](https://md.research.elamur.ai/free/en/papers/durmus-2019-role-pragmatic-discourse-context-determining/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) ## References 23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Emiel Krahmer and Kees Van Deemter. 2012. Computational generation of referring expressions: A survey. Computational Linguistics, 38(1):173–218. [↗](https://doi.org/10.1162/coli_a_00088) [our card](https://md.research.elamur.ai/free/en/papers/krahmer-2012-computational-generation-referring-expressions-survey/) 2. Harsh Agrawal, Karan Desai, Xinlei Chen, Rishabh Jain, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2018. nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658. [↗](https://arxiv.org/abs/1812.08658) 3. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73. [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/) 4. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936–945, Copenhagen, Denmark. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/anderson-2017-guided-open-vocabulary-image-captioning/) 5. Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In IEEE Computer Vision and Pattern Recognition, pages 951–958. IEEE. [find](https://scholar.google.com/scholar?q=Christoph+H+Lampert%2C+Hannes+Nickisch%2C+and+Stefan+Harmeling%2E+2009%2E+Learning+to+detect+unseen+object+classes+by+between%2Dclass+attribute+transfer%2E+In+IEEE+Computer+Vision+and+Pattern+Recognition%2C+pages+9) 6. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Austin, Texas. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d16-1125) [our card](https://md.research.elamur.ai/free/en/papers/andreas-2016-reasoning-about-pragmatics-neural-listeners/) 7. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C.Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, volume 8693, pages 740–755. Springer International Publishing. [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/) 8. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. [↗](https://arxiv.org/abs/1504.00325) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/) 9. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In CVPR 2016. [our card](https://md.research.elamur.ai/free/en/papers/mao-2015-generation-comprehension-unambiguous-object-descriptions/) 10. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22:1– 39. [find](https://scholar.google.com/scholar?q=Herbert+H%2E+Clark+and+Deanna+Wilkes%2DGibbs%2E+1986%2E+Referring+as+a+collaborative+process%2E+Cognition%2C+22%3A1%E2%80%93+39%2E) 11. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. [↗](https://arxiv.org/abs/1409.1556) [our card](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/) 12. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 439–443. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/gordon-2018-pragmatically-informative-image-captioning-character/) 13. Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in neural information processing systems, pages 935–943. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Milind+Ganjoo%2C+Christopher+D+Manning%2C+and+Andrew+Ng%2E+2013%2E+Zero%2Dshot+learning+through+cross%2Dmodal+transfer%2E+In+Advances+in+neural+information+processing+systems%2C+pages+935%E2%80%93943%2E) 14. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Computer Vision and Pattern Recognition (CVPR), volume 3. [find](https://scholar.google.com/scholar?q=Ramakrishna+Vedantam%2C+Samy+Bengio%2C+Kevin+Murphy%2C+Devi+Parikh%2C+and+Gal+Chechik%2E+2017%2E+Context%2Daware+captions+from+context%2Dagnostic+supervision%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+volume) 15. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science, 19(2):233–263. [↗](https://doi.org/10.1207/s15516709cog1902_3) [our card](https://md.research.elamur.ai/free/en/papers/dale-1995-computational-interpretations-gricean-maxims-generation/) 16. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09. [find](https://scholar.google.com/scholar?q=J%2E+Deng%2C+W%2E+Dong%2C+R%2E+Socher%2C+L%2E%2DJ%2E+Li%2C+K%2E+Li%2C+and+L%2E+Fei%2DFei%2E+2009%2E+ImageNet%3A+A+Large%2DScale+Hierarchical+Image+Database%2E+In+CVPR09%2E) 17. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition. [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/) 18. Rui Fang, Malcolm Doering, and Joyce Y Chai. 2014. Collaborative models for referring expression generation in situated dialogue. In Twenty-Eighth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Rui+Fang%2C+Malcolm+Doering%2C+and+Joyce+Y+Chai%2E+2014%2E+Collaborative+models+for+referring+expression+generation+in+situated+dialogue%2E+In+Twenty%2DEighth+AAAI+Conference+on+Artificial+Intelligence%2E) 19. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. Modeling Context in Referring Expressions, pages 69–85. Springer International Publishing, Cham. [our card](https://md.research.elamur.ai/free/en/papers/yu-2016-modeling-context-referring-expressions/) 20. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. [↗](https://doi.org/10.1126/science.1218633) [our card](https://md.research.elamur.ai/free/en/papers/frank-2012-predicting-pragmatic-reasoning-language-games/) 21. Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speakerlistener-reinforcer model for referring expressions. In Computer Vision and Pattern Recognition (CVPR), volume 2. [find](https://scholar.google.com/scholar?q=Licheng+Yu%2C+Hao+Tan%2C+Mohit+Bansal%2C+and+Tamara+L+Berg%2E+2017%2E+A+joint+speakerlistener%2Dreinforcer+model+for+referring+expressions%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+volume+2%2E) 22. Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. 2016. Deep compositional captioning: Describing novel object categories without paired training data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10. [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Subhashini+Venugopalan%2C+Marcus+Rohrbach%2C+Raymond+Mooney%2C+Kate+Saenko%2C+and+Trevor+Darrell%2E+2016%2E+Deep+compositional+captioning%3A+Describing+novel+object+categories+without+paired+tr) 23. Sina Zarrieß and David Schlangen. 2018. Decoding strategies for neural referring expression generation. Proceedings of INLG 2018. [find](https://scholar.google.com/scholar?q=Sina+Zarrie%C3%9F+and+David+Schlangen%2E+2018%2E+Decoding+strategies+for+neural+referring+expression+generation%2E+Proceedings+of+INLG+2018%2E)