--- title: "“Caption” as a Coherence Relation — Alikhani 2019" description: "The paper asks whether standard image–caption datasets can support general models that connect pictures and language." authors: ["Malihe Alikhani", "Matthew Stone"] year: 2019 venue: "Proceedings of the Second Workshop on Shortcomings in Vision and Language" source: "https://aclanthology.org/W19-1806/" doi: "10.18653/v1/w19-1806" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/alikhani-2019-caption-as-coherence-relation-evidence/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/alikhani-2019-caption-as-coherence-relation-evidence/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/page/4/) / “Caption” as a Coherence Relation: Evidence and Implications # “Caption” as a Coherence Relation: Evidence and Implications **Authors:** [Malihe Alikhani](https://md.research.elamur.ai/free/authors/malihe-alikhani/), [Matthew Stone](https://md.research.elamur.ai/free/authors/matthew-stone/) · **Year:** 2019 · **Venue:** Proceedings of the Second Workshop on Shortcomings in Vision and Language · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-1806/) · [PDF](https://aclanthology.org/W19-1806.pdf) · [DOI 10.18653/v1/w19-1806](https://doi.org/10.18653/v1/w19-1806) · [OpenAlex](https://openalex.org/W2964127652) **Citations:** 25 citations (openalex-snapshot · 2026-10-14) · [23](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper asks whether standard image–caption datasets can support general models that connect pictures and language. The authors argue that they cannot, because captioning is a specific coherence relation in the sense of Hobbs (1990), with its own limits on content and perspective. They compare verb usage across five image–text corpora: crowdsourced COCO, Flickr30K and VIST, plus Conceptual Captions and the Recipe dataset, which pair images with text written for other purposes. The reference points are ANC and Facebook children's stories. Verbs, tense and aspect were extracted with SpaCy-based rules, and 500 sentences per corpus were hand-annotated for lexical aspect. Caption corpora use few verbs, and those verbs are repetitive. They favour present progressive forms, visible events and atelic or stative descriptions. Models trained on COCO inherit this narrow pattern. The Recipe data, where text and images illustrate each other, looks much more like ordinary English. The authors conclude that retrieval, illustrated-document generation and grounded semantics need more naturally occurring data, and that the different text–image relations should be modelled explicitly. ## Key points - **Verb density:** Verbs make up 0.184 of ANC tokens. The caption corpora fall well below this, with CC at 0.065, COCO at 0.026, VIST at 0.017 and Flickr at just 0.012. - **Verb concentration:** The 10 most frequent verb lemmas account for about 0.60 of verb tokens in COCO and Flickr. The figure is 0.443 in ANC and 0.392 in Recipe. - **Frequent Caption Verbs (FCVs):** The authors list 17 such verbs, e.g. *sitting*, *holding*, *wearing*. They make up 0.535–0.638 of the verbs output by eight captioning models trained on COCO. - **Telicity:** Telic verbs are scarce in Flickr and COCO, at about 28–29 per 500 sentences. Recipe and ANC have about 300 each, and the CC telic rate is 0.252. Flickr≈COCO and Recipe≈ANC, while all other pairs differ significantly. Annotator agreement was Cohen's κ = 0.77. - **Atelicity is systematic:** In Flickr and COCO, FCVs yield atelic readings 96% of the time, compared with 39% in ANC. The annotations are released on GitHub (malihealikhani/Captions). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/) ## Cited by Cited by 23 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 19 not yet. - [CITE: A Corpus of Image-Text Discourse Relations](https://md.research.elamur.ai/free/papers/alikhani-2019-cite-corpus-image-text-discourse/) - [Human Temporal Inferences Go Beyond Aspectual Class](https://md.research.elamur.ai/free/papers/prus-2024-human-temporal-inferences-go-beyond/) - [GePaDeSE: A New Resource for Clause-Level Aspect in German Parliamentary Debates](https://md.research.elamur.ai/free/papers/schlenker-2026-gepadese-new-resource-clause-level/) - [A Corpus for Reasoning About Natural Language Grounded in Photographs (NLVR2)](https://md.research.elamur.ai/free/papers/suhr-2019-corpus-reasoning-natural-language-grounded/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{alikhani2019caption, title = {{“Caption” as a Coherence Relation: Evidence and Implications}}, author = {Alikhani, Malihe and Stone, Matthew}, booktitle = {Proceedings of the Second Workshop on Shortcomings in Vision and Language}, year = {2019}, doi = {10.18653/v1/w19-1806}, url = {https://doi.org/10.18653/v1/w19-1806} } ``` - [← previous Zero-shot transfer for implicit discourse relation classification](https://md.research.elamur.ai/free/papers/kurfal-2019-zero-shot-transfer-implicit-discourse/) - [next → “Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer](https://md.research.elamur.ai/free/papers/sudhakar-2019-transforming-delete-retrieve-generate-approach/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/page/4/) ## References 6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 86–90. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Collin+F+Baker%2C+Charles+J+Fillmore%2C+and+John+B+Lowe%2E+1998%2E+The+berkeley+framenet+project%2E+In+Proceedings+of+the+17th+international+conference+on+Computational+linguistics%2DVolume+1%2C+pages+86%E2%80%9390%2E+Associ) [our card](https://md.research.elamur.ai/free/papers/lex-1998-baker-berkeley-framenet/) 2. Raffaella Bernardi, Ruket Çakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2017. Automatic description generation from images: A survey of models, datasets, and evaluation measures (extended abstract). In Proceedings of the Twe… [find](https://scholar.google.com/scholar?q=Raffaella+Bernardi%2C+Ruket+C%CC%A7akici%2C+Desmond+Elliott%2C+Aykut+Erdem%2C+Erkut+Erdem%2C+Nazli+Ikizler%2DCinbis%2C+Frank+Keller%2C+Adrian+Muscat%2C+and+Barbara+Plank%2E+2017%2E+Automatic+description+generation+from+images%3A+) 3. Neil Cohn. 2013. Visual narrative structure. Cognitive science, 37(3):413–452. [find](https://scholar.google.com/scholar?q=Neil+Cohn%2E+2013%2E+Visual+narrative+structure%2E+Cognitive+science%2C+37%283%29%3A413%E2%80%93452%2E) 4. Bernard Comrie. 1976. Aspect: An introduction to the study of verbal aspect and related problems, volume 2. Cambridge university press. [find](https://scholar.google.com/scholar?q=Bernard+Comrie%2E+1976%2E+Aspect%3A+An+introduction+to+the+study+of+verbal+aspect+and+related+problems%2C+volume+2%2E+Cambridge+university+press%2E) 5. Samuel Cumming, Gabriel Greenberg, and Rory Kelly. 2017. Conventions of viewpoint coherence in film. Philosophers’ Imprint, 17(1):1–29. [find](https://scholar.google.com/scholar?q=Samuel+Cumming%2C+Gabriel+Greenberg%2C+and+Rory+Kelly%2E+2017%2E+Conventions+of+viewpoint+coherence+in+film%2E+Philosophers%E2%80%99+Imprint%2C+17%281%29%3A1%E2%80%9329%2E) 6. Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. 2017. Towards diverse and natural image descriptions via a conditional gan. In The IEEE International Conference on Computer Vision (ICCV). [find](https://scholar.google.com/scholar?q=Bo+Dai%2C+Sanja+Fidler%2C+Raquel+Urtasun%2C+and+Dahua+Lin%2E+2017%2E+Towards+diverse+and+natural+image+descriptions+via+a+conditional+gan%2E+In+The+IEEE+International+Conference+on+Computer+Vision+%28ICCV%29%2E)