---
title: "Conceptual Captions — Sharma 2018"
description: "Image captioning models had been trained mostly on MS-COCO. 23 references, cited by 474 papers here."
authors: ["Piyush Sharma", "Nan Ding", "Sebastian Goodman", "Radu Soricut"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P18-1238/"
doi: "10.18653/v1/p18-1238"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

# Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning

**Authors:** Piyush Sharma, Nan Ding, [Sebastian Goodman](https://md.research.elamur.ai/free/en/authors/sebastian-goodman/) and [Radu Soricut](https://md.research.elamur.ai/free/en/authors/radu-soricut/) · **Year:** 2018 · **Venue:** Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-1238/) · [PDF](https://aclanthology.org/P18-1238.pdf) · [DOI 10.18653/v1/p18-1238](https://doi.org/10.18653/v1/p18-1238) · [OpenAlex](https://openalex.org/W2886641317)

**Citations:** 1,804 citations (OpenAlex, 2026-10-14) · [474](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Image captioning models had been trained mostly on MS-COCO. That dataset is hand-curated and fairly small for deep networks, at around 120K images. The authors address this with Conceptual Captions, a dataset of about 3.3M image–caption pairs harvested automatically from the web. A Flume pipeline started from the Alt-text attributes of over 5 billion images on roughly 1 billion English webpages. It filtered the candidates by image properties, by text quality and by checking that image and text agree. It then rewrote the surviving captions through hypernymization: proper names, places and dates were replaced with general terms (for example, actor names became "actors") or removed, using the Google Knowledge Graph. The result covers a wider range of images and caption styles than COCO, such as product shots, cartoons and drawings. Captioning models were built on Inception-ResNet-v2 image features, paired with either an RNN or a Transformer decoder. In human evaluation, models trained on Conceptual Captions did better and hallucinated objects less often than COCO-trained ones. Transformer decoders also beat RNN decoders. Standard automatic metrics often disagreed with these human judgments, which led the authors to question how reliable those metrics are.

## Key points
- Scale: about 3.3M pairs, roughly ten times as many images as COCO. Only about 0.2% of the original web candidates made it through every filter. The dataset is published at github.com/google-research-datasets/conceptual-captions.
- Filtering stages:
  - **Image filter:** keeps only JPEGs with both sides over 400 px and an aspect ratio of at most 2, and removes offensive images. This drops more than 65% of candidates.
  - **Text heuristics:** use Google Cloud Natural Language annotations (POS tags, sentiment, capitalization, a Wikipedia-based vocabulary) and let only about 3% through.
  - **Image–text match:** compares Cloud Vision labels (on the order of 10^5 classes) with the caption words and removes about 60% of what remains.
- Hypernymization combines named-entity recognition, dependency parses and Knowledge Graph hypernyms. It removes details that a model could not learn from pixels alone, such as who a person is or where a photo was taken.
- The best architecture was T2T8x8, which uses Inception-ResNet-v2 features with a Transformer decoder. The text reports a CIDEr score of up to 1.468 on the 22.5K-image Conceptual Captions test set, although the extracted Table 6 shows higher values (up to 1.676). I couldn't resolve this because part of the paper was missing from the extracted text.
- On COCO and Flickr test sets, CIDEr, ROUGE-L, METEOR and SPICE favored COCO-trained and RNN models, which contradicts the human ratings. The authors argue these metrics do not penalize hallucinated content enough.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/)

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 272 works · 255,532 citations

## Cite

Cite as: Piyush Sharma, Nan Ding, Sebastian Goodman and Radu Soricut. 2018. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sharma2018conceptual,
  title = {{Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning}},
  author = {Piyush Sharma, Nan Ding, Sebastian Goodman and Radu Soricut},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1238},
  url = {https://doi.org/10.18653/v1/p18-1238}
}
```

- [← previous Relational inductive biases, deep learning, and graph networks](https://md.research.elamur.ai/free/en/papers/battaglia-2018-relational-inductive-biases-deep-learning/)
- [next → FEVER: a Large-scale Dataset for Fact Extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

23 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John Platt, et al. 2015. From captions to visual concepts and back. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Hao+Fang%2C+Saurabh+Gupta%2C+Forrest+Iandola%2C+Rupesh+Srivastava%2C+Li+Deng%2C+Piotr+Dolla%CC%81r%2C+Jianfeng+Gao%2C+Xiaodong+He%2C+Margaret+Mitchell%2C+John+Platt%2C+et+al%2E+2015%2E+From+captions+to+visual+concepts+and+back%2E+I)
2. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic propositional image caption evaluation. In ECCV. [our card](https://md.research.elamur.ai/free/en/papers/anderson-2016-spice-semantic-propositional-image-caption/)
3. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation+9%288%29%3A1735%E2%80%93+1780%2E)
4. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
5. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR . [find](https://scholar.google.com/scholar?q=Micah+Hodosh%2C+Peter+Young%2C+and+Julia+Hockenmaier%2E+2013%2E+Framing+image+description+as+a+ranking+task%3A+Data%2C+models+and+evaluation+metrics%2E+JAIR+%2E)
6. Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy. 2016. Speed/accuracy trade-offs for modern convolutional object detectors. CoRR abs/1611.10012. — [arXiv:1611.10012](https://arxiv.org/abs/1611.10012) [find](https://scholar.google.com/scholar?q=Jonathan+Huang%2C+Vivek+Rathod%2C+Chen+Sun%2C+Menglong+Zhu%2C+Anoop+Korattikara%2C+Alireza+Fathi%2C+Ian+Fischer%2C+Zbigniew+Wojna%2C+Yang+Song%2C+Sergio+Guadarrama%2C+and+Kevin+Murphy%2E+2016%2E+Speed%2Faccuracy+trade%2Doffs+for)
7. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
8. Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2015. Unifying visual-semantic embeddings with multimodal neural language models. Transactions of the Association for Computational Linguistics . [our card](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/)
9. A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=A%2E+Krizhevsky%2C+I%2E+Sutskever%2C+and+G%2E+Hinton%2E+2012%2E+Imagenet+classification+with+deep+convolutional+neural+networks%2E+In+NIPS%2E)
10. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of ACL. [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/)
11. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. CoRR abs/1405.0312. [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
12. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [our card](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/)
13. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015a. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3156–3164. [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
14. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015b. Show and tell: A neural image caption generator. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
15. Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proc. of the 32nd International Conference on Machine Learning (ICML). [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
16. Z. Yang, Y. Yuan, Y. Wu, R. Salakhutdinov, and W. W. Cohen. 2016. Review networks for caption generation. In NIPS. [find](https://scholar.google.com/scholar?q=Z%2E+Yang%2C+Y%2E+Yuan%2C+Y%2E+Wu%2C+R%2E+Salakhutdinov%2C+and+W%2E+W%2E+Cohen%2E+2016%2E+Review+networks+for+caption+generation%2E+In+NIPS%2E)
17. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL 2:67–78. [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+2014%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+TACL+2%3A67%E2%80%9378%2E)
18. Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. 2017. Optimization of image description metrics using policy gradient methods. In International Conference on Computer Vision (ICCV). [find](https://scholar.google.com/scholar?q=Siqi+Liu%2C+Zhenhai+Zhu%2C+Ning+Ye%2C+Sergio+Guadarrama%2C+and+Kevin+Murphy%2E+2017%2E+Optimization+of+image+description+metrics+using+policy+gradient+methods%2E+In+International+Conference+on+Computer+Vision+%28ICCV)
19. Junhua Mao, Jiajing Xu, Yushi Jing, and Alan Yuille. 2016. Training and evaluating multimodal word embeddings with large-scale web annotated images. In NIPS. [find](https://scholar.google.com/scholar?q=Junhua+Mao%2C+Jiajing+Xu%2C+Yushi+Jing%2C+and+Alan+Yuille%2E+2016%2E+Training+and+evaluating+multimodal+word+embeddings+with+large%2Dscale+web+annotated+images%2E+In+NIPS%2E)
20. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. CoRR abs/1511.06732. [our card](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
21. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2E+pages+3104%E2%80%933112%2E)
22. Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR abs/1602.07261. — [arXiv:1602.07261](https://arxiv.org/abs/1602.07261) [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Sergey+Ioffe%2C+and+Vincent+Vanhoucke%2E+2016%2E+Inception%2Dv4%2C+inception%2Dresnet+and+the+impact+of+residual+connections+on+learning%2E+CoRR+abs%2F1602%2E07261%2E+%E2%80%94+%5BarXiv%3A1602%2E07261%5D%28https%3A%2F%2Farxiv)
23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)