---
title: "Pragmatic Factors in Image Description: The Case of Negations"
description: "The paper looks at why crowdworkers sometimes use negation when describing photos, even though… 20 references, cited by 12 papers here."
authors: ["Emiel van Miltenburg", "Roser Morante", "Desmond Elliott"]
year: 2016
venue: "Proceedings of the 5th Workshop on Vision and Language"
source: "https://aclanthology.org/W16-3207/"
doi: "10.18653/v1/w16-3207"
arxiv: "1606.06164"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/miltenburg-2016-pragmatic-factors-image-description-case/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/miltenburg-2016-pragmatic-factors-image-description-case/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Pragmatic Factors in Image Description: The Case of Negations

# Pragmatic Factors in Image Description: The Case of Negations

**Authors:** Emiel van Miltenburg, [Roser Morante](https://md.research.elamur.ai/free/en/authors/roser-morante/) and [Desmond Elliott](https://md.research.elamur.ai/free/en/authors/desmond-elliott/) · **Year:** 2016 · **Venue:** Proceedings of the 5th Workshop on Vision and Language · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W16-3207/) · [PDF](https://aclanthology.org/W16-3207.pdf) · [DOI 10.18653/v1/w16-3207](https://doi.org/10.18653/v1/w16-3207) · [arXiv 1606.06164](https://arxiv.org/abs/1606.06164) · [OpenAlex](https://openalex.org/W2464060482)

**Citations:** 20 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/miltenburg-2016-pragmatic-factors-image-description-case/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper looks at why crowdworkers sometimes use negation when describing photos, even though they were told to describe only what is visible. The authors searched Flickr30K captions for non-affixal negators and implicit negatives using regular expressions. The implicit negatives were *without*, *sans*, *minus* and verbs such as *lack* and *miss*. They ran the same search on MS COCO. Negation turned out to be rare but stable: about half a percent of captions in both corpora. Usually only one of an image's five descriptions contains it, which suggests it is a subjective choice.

The authors read all Flickr30K hits twice, first to build a taxonomy of negation uses and then to apply it. They then double-annotated the corpus with these categories and reached moderate agreement. Most negations mark a salient absence, especially missing clothing. The authors argue that negation reflects a gap between expectation and reality. Generating it would therefore require background knowledge: norms, scripts, action recognition, gaze and scene types. This makes negation a useful window into subjective language in vision-and-language data.

## Key points
- **Flickr30K search results:**
  - 896 sentences with negations, 892 of them unique, including 31 false positives.
  - The most frequent terms were *no* (371), *not* (198), *without* (141), *miss* (69) and *n't* (68).
- **Comparison across corpora:**
  - MS COCO train and validation splits gave 3339 negated sentences, 3232 of them unique.
  - The negation rate was 0.56% in Flickr30K and 0.54% in MS COCO, so the phenomenon looks robust across datasets.
  - A single negated description per image was the norm: 86.25% of cases in Flickr30K and 72.05% in MS COCO.
- **Taxonomy:** salient absence; negation of action/behavior; negation of property; negation of attitude; outside the frame; (preventing) future events; quotes and idioms; other. A Meta class covers comments about the task itself.
- **Annotation:**
  - Two annotators reached Cohen's κ=0.67, with 77% raw agreement.
  - Salient absence was by far the largest category, with 488 sentences.
  - 39 sentences also used negation to set up a contrast.
- **Clothing skew:** 282 negations (32%) are about shirtlessness and 59 (7%) about missing shoes. This skew may make it hard for systems to learn other kinds of negation. Code, data and guidelines are released on GitHub (annotating-negations).

## Interesting ideas
- Treat a negation as evidence that the speaker held an expectation which the scene broke, as in a graduate without a cap; mined at scale, such sentences expose default script knowledge.
- A negated part of a set licenses an inference about the rest: "two of three without shoes" implies the third wears shoes, a scalar implicature drawn from the negation.
- Separate negations that are mentioned (tattoo text, game names, idioms like "path to nowhere") from negations that are actually asserted about the scene.
- Compare several independent descriptions of the same item to tell objective content from subjective choices: if only one describer uses a construction, it is optional.
- A negation sounds natural only when it cancels a plausible expectation, which is why "the Queen isn't wearing jeans" is odd.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): negated clauses in texts could be harvested as candidate defaults for frames and scripts, e.g. "graduation → cap", and stored as typical-but-cancellable properties in the knowledge graph.
- **I2** (frames): the implicature module could add a rule that a negated subset quantifier asserts the complementary property for the remaining members of the group.
- **I3** (frames): quoted or idiomatic negators should be flagged so they do not flip the polarity of a frame or delete a state in the world of the text.
- **I4** (learn): when witness parsers or LLM teacher tiers describe the same text, content produced by only one of them could be marked as subjective or low-confidence instead of being merged into silver data.
- **I5** (back): the generator should emit a negative statement only when the world model holds a default that the current state contradicts; otherwise it should leave the absence unmentioned.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 89 works · 81,337 citations
- [Vrije Universiteit Amsterdam](https://md.research.elamur.ai/free/en/universities/vrije-universiteit-amsterdam/) — University, Amsterdam, Netherlands · 33 works · 3,294 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Emiel van Miltenburg, Roser Morante and Desmond Elliott. 2016. Pragmatic Factors in Image Description: The Case of Negations. Proceedings of the 5th Workshop on Vision and Language.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{miltenburg2016pragmatic,
  title = {{Pragmatic Factors in Image Description: The Case of Negations}},
  author = {Emiel {van Miltenburg}, Roser Morante and Desmond Elliott},
  booktitle = {Proceedings of the 5th Workshop on Vision and Language},
  year = {2016},
  doi = {10.18653/v1/w16-3207},
  eprint = {1606.06164},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w16-3207}
}
```

- [← previous Left Dislocation in North-Eastern Neo-Aramaic dialects](https://md.research.elamur.ai/free/en/papers/khan-2016-left-dislocation-north-eastern-neo/)
- [next → Tweet Acts: A Speech Act Classifier for Twitter](https://md.research.elamur.ai/free/en/papers/vosoughi-2016-tweet-acts-speech-act-classifier/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. [The berkeley framenet project](https://md.research.elamur.ai/free/en/papers/baker-1998-berkeley-framenet/). In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics -Volume 1, ACL ’98, pages 86–90, Stroudsburg, PA, USA. Association for Computational Linguistics.
2. Camiel J Beukeboom, Catrin Finkenauer, and Daniël HJ Wigboldus. 2010. The negation bias: when negations signal stereotypic expectancies. Journal of personality and social psychology, 99(6):978. [↗](https://doi.org/10.1037/a0020861)
3. Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL ’09, pages 602–610, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nathanael+Chambers+and+Dan+Jurafsky%2E+2009%2E+Unsupervised+learning+of+narrative+schemas+and+their+participants%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+I)
4. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. [Microsoft COCO captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/). CoRR, abs/1504.00325. [↗](https://arxiv.org/abs/1504.00325)
5. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. [Multi30K: Multilingual English-German Image Descriptions](https://md.research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/). CoRR, abs/1605.00459. [↗](https://arxiv.org/abs/1605.00459)
6. Ethan Fast, William McGrath, Pranav Rajpurkar, and Michael S. Bernstein. 2016. Augur: Mining human behaviors from fiction to power interactive systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, pages 237–247, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Ethan+Fast%2C+William+McGrath%2C+Pranav+Rajpurkar%2C+and+Michael+S%2E+Bernstein%2E+2016%2E+Augur%3A+Mining+human+behaviors+from+fiction+to+power+interactive+systems%2E+In+Proceedings+of+the+2016+CHI+Conference+on+Hum)
7. Charles J Fillmore. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences, 280(1):20–32. [↗](https://doi.org/10.1111/j.1749-6632.1976.tb25467.x)
8. James J. Gibson. 1977. The theory of affordances. In R. E. Shaw and J. Bransford, editors, Perceiving, Acting, and Knowing. Hillsdale, NJ: Lawrence Erlbaum Associates. [↗](https://doi.org/10.4324/9781315740218-18)
9. Laurence R. Horn. 1972. On the Semantic Properties of Logical Operators in English. Ph.D. thesis, UCLA, Los Angeles. [find](https://scholar.google.com/scholar?q=Laurence+R%2E+Horn%2E+1972%2E+On+the+Semantic+Properties+of+Logical+Operators+in+English%2E+Ph%2ED%2E+thesis%2C+UCLA%2C+Los+Angeles%2E)
10. Laurence R. Horn. 1989. A natural history of negation. CSLI Publications. [find](https://scholar.google.com/scholar?q=Laurence+R%2E+Horn%2E+1989%2E+A+natural+history+of+negation%2E+CSLI+Publications%2E)
11. Geoffrey Leech. 1983. Principles of pragmatics. London and New York: Longman. [find](https://scholar.google.com/scholar?q=Geoffrey+Leech%2E+1983%2E+Principles+of+pragmatics%2E+London+and+New+York%3A+Longman%2E)
12. Aude Oliva and Antonio Torralba. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42(3):145–175. [↗](https://doi.org/10.1023/a:1011139631724)
13. Marco Pennacchiotti, Diego De Cao, Roberto Basili, Danilo Croce, and Michael Roth. 2008. [Automatic induction of framenet lexical units](https://md.research.elamur.ai/free/en/papers/pennacchiotti-2008-automatic-induction-framenet-lexical-units/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 457–465. Association for Computational Linguistics.
14. Ronald Poppe. 2010. A survey on vision-based human action recognition. Image and Vision Computing, 28(6):976 – 990. [↗](https://doi.org/10.1016/j.imavis.2009.11.014)
15. Roger C. Schank and Robert P. Abelson. 1977. Scripts, Plans, Goals, and Understanding: An Inquiry Into Human Knowledge Structures. L. Erlbaum Associates. [find](https://scholar.google.com/scholar?q=Roger+C%2E+Schank+and+Robert+P%2E+Abelson%2E+1977%2E+Scripts%2C+Plans%2C+Goals%2C+and+Understanding%3A+An+Inquiry+Into+Human+Knowledge+Structures%2E+L%2E+Erlbaum+Associates%2E)
16. Antonio Torralba, Aude Oliva, Monica S Castelhano, and John M Henderson. 2006. Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological review, 113(4):766. [↗](https://doi.org/10.1037/0033-295x.113.4.766)
17. Gunnel Tottie. 1980. Affixal and non-affixal negation in English: Two systems in (almost) complementary distribution. Studia linguistica, 34(2):101–123. [↗](https://doi.org/10.1111/j.1467-9582.1980.tb00310.x)
18. Roberto Valenti, Nicu Sebe, and Theo Gevers. 2012. Combining head pose and eye location information for gaze estimation. IEEE Transactions on Image Processing, 21(2):802–815, Feb. [↗](https://doi.org/10.1109/tip.2011.2162740)
19. Emiel van Miltenburg. 2016. Stereotyping and bias in the Flickr30K dataset. In Proceedings of the 11 th Workshop on Multimodal Corpora (MMC2016), pages 1–4. [↗](https://doi.org/10.48550/arxiv.1605.06083)
20. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67– 78. [↗](https://doi.org/10.1162/tacl_a_00166)

*Updated 24 October 2026*