---
title: "Object Hallucination in Image Captioning — Rohrbach 2018"
description: "Modern image captioning systems often mention objects that are absent from the picture."
authors: ["Anna Rohrbach", "Lisa Anne Hendricks", "Kaylee Burns", "Trevor Darrell", "Kate Saenko"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1437/"
doi: "10.18653/v1/d18-1437"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Object Hallucination in Image Captioning

# Object Hallucination in Image Captioning

**Authors:** [Anna Rohrbach](https://md.research.elamur.ai/free/en/authors/anna-rohrbach/), [Lisa Anne Hendricks](https://md.research.elamur.ai/free/en/authors/lisa-anne-hendricks/), Kaylee Burns, [Trevor Darrell](https://md.research.elamur.ai/free/en/authors/trevor-darrell/), [Kate Saenko](https://md.research.elamur.ai/free/en/authors/kate-saenko/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1437/) · [PDF](https://aclanthology.org/D18-1437.pdf) · [DOI 10.18653/v1/d18-1437](https://doi.org/10.18653/v1/d18-1437) · [OpenAlex](https://openalex.org/W2892087592)

**Citations:** 403 citations (openalex-snapshot · 2026-10-14) · [214](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Modern image captioning systems often mention objects that are absent from the picture. Standard metrics compare generated text only with reference captions, so they can miss this error. To catch it, the authors propose a metric that compares every object a caption names against the 80 MSCOCO categories, and they call it CHAIR, short for "Caption Hallucination Assessment with Image Relevance." Ground truth comes from both MSCOCO segmentation labels and MSCOCO Captions, with synonym mapping and handling of compounds like "hot dog". CHAIR has two variants: CHAIRi measures the share of mentioned objects that are hallucinated, and CHAIRs measures the share of sentences containing at least one hallucination.

They apply CHAIR to many models with different architectures (FC, LRCN, Att2In, TopDown, TopDown-BB, NBT, GAN) and training losses (cross-entropy, self-critical). To trace causes, they compare errors with an image-only classifier and an LSTM language model. Models that hallucinate more tend to produce errors that agree with language priors. Top scores on standard metrics do not guarantee fewer hallucinations. The work argues that captioning should be evaluated and developed with image relevance in mind.

## Key points
- Current models hallucinate between 5.5% and 13.1% of mentioned MSCOCO objects. Self-critical (CIDEr-optimizing) training raises CIDEr but also increases hallucination.
- Attention reduces hallucination, but most of the gain seems to come from access to convolutional features rather than from the attention mechanism itself. Models with stronger image consistency hallucinate less.
- Among standard metrics, SPICE correlates best with hallucination-free output (0.27 for FC vs. about 0.2 for METEOR and CIDEr). Even so, models with similar scores can differ noticeably in hallucination rates.
- In an AMT study, workers rated captions from 5 models on 500 images, with 3 annotators per image. Adding 1−CHAIR to a sentence metric raised correlation with human scores, for example CIDEr from 0.282 to 0.322.
- A hallucinated word changes the subsequent words 47.3% of the time, and one hallucination can trigger another (e.g., "dog" leading to "frisbee").

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

214 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 190 not yet.

- [ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation](https://md.research.elamur.ai/free/en/papers/abdaljalil-2026-imageeval-2026-culturally-grounded-arabic/)
- [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://md.research.elamur.ai/free/en/papers/baek-2023-knowledge-augmented-language-model-prompting/)
- [Logical Natural Language Generation from Open-Domain Tables](https://md.research.elamur.ai/free/en/papers/chen-2020-logical-natural-language-generation-open/)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://md.research.elamur.ai/free/en/papers/chen-2023-minigpt-v2-large-language-model/)
- [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://md.research.elamur.ai/free/en/papers/dhingra-2019-handling-divergent-reference-texts-when/)
- [Semantic Noise Matters for Neural Natural Language Generation](https://md.research.elamur.ai/free/en/papers/dusek-2019-semantic-noise-matters-neural-natural/)
- [Multilingual Sequence-to-Sequence Models for Hebrew NLP](https://md.research.elamur.ai/free/en/papers/findings-2023-eyal-multilingual-sequence-sequence-models-hebrew/)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/)
- [TRUE: Re-evaluating Factual Consistency Evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/)
- [A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://md.research.elamur.ai/free/en/papers/liu-2022-token-level-reference-free-hallucination/)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/)
- [Reasoning with Language Model Prompting: A Survey](https://md.research.elamur.ai/free/en/papers/qiao-2022-reasoning-language-model-prompting-survey/)
- [A Survey of Hallucination in Large Foundation Models](https://md.research.elamur.ai/free/en/papers/rawte-2023-survey-hallucination-large-foundation-models/)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/)
- [Fine-tuning Language Models for Factuality](https://md.research.elamur.ai/free/en/papers/tian-2023-fine-tuning-language-models-factuality/)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://md.research.elamur.ai/free/en/papers/xiao-2021-hallucination-predictive-uncertainty-conditional-language/)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/)
- [How Language Model Hallucinations Can Snowball](https://md.research.elamur.ai/free/en/papers/zhang-2023-language-model-hallucinations-snowball/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rohrbach2018object,
  title = {{Object Hallucination in Image Captioning}},
  author = {Rohrbach, Anna and Hendricks, Lisa Anne and Burns, Kaylee and Darrell, Trevor and Saenko, Kate},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1437},
  url = {https://doi.org/10.18653/v1/d18-1437}
}
```

- [← previous Normalizing Early English Letters to Present-day English Spelling](https://md.research.elamur.ai/free/en/papers/hamalainen-2018-normalizing-early-english-letters-present/)
- [next → Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words](https://md.research.elamur.ai/free/en/papers/mohammad-2018-obtaining-reliable-human-ratings-valence/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision, pages 382–398. Springer. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Basura+Fernando%2C+Mark+Johnson%2C+and+Stephen+Gould%2E+2016%2E+Spice%3A+Semantic+propositional+image+caption+evaluation%2E+In+European+Conference+on+Computer+Vision%2C+pages+382%E2%80%93398%2E+Springer%2E) [our card](https://md.research.elamur.ai/free/en/papers/anderson-2016-spice-semantic-propositional-image-caption/)
2. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and vqa. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Xiaodong+He%2C+Chris+Buehler%2C+Damien+Teney%2C+Mark+Johnson%2C+Stephen+Gould%2C+and+Lei+Zhang%2E+2018%2E+Bottom%2Dup+and+top%2Ddown+attention+for+image+captioning+and+vqa%2E+In+Proceedings+of+the+IEEE+Co)
3. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. metrics for image captio… [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+Meteor%3A+An+automatic+metric+for+mt+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+ACL+Workshop+on+Intrinsic+and+Extrinsic+Eva) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
4. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32… [find](https://scholar.google.com/scholar?q=Ranjay+Krishna%2C+Yuke+Zhu%2C+Oliver+Groth%2C+Justin+Johnson%2C+Kenji+Hata%2C+Joshua+Kravitz%2C+Stephanie+Chen%2C+Yannis+Kalantidis%2C+Li%2DJia+Li%2C+David+A+Shamma%2C+et+al%2E+2017%2E+Visual+genome%3A+Connecting+language+and+vi) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
5. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer. [find](https://scholar.google.com/scholar?q=Tsung%2DYi+Lin%2C+Michael+Maire%2C+Serge+Belongie%2C+James+Hays%2C+Pietro+Perona%2C+Deva+Ramanan%2C+Piotr+Dolla%CC%81r%2C+and+C+Lawrence+Zitnick%2E+2014%2E+Microsoft+coco%3A+Common+objects+in+context%2E+In+European+conference+on+) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
6. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018. Neural baby talk. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Jiasen+Lu%2C+Jianwei+Yang%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+2018%2E+Neural+baby+talk%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2E)
7. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. [find](https://scholar.google.com/scholar?q=Xinlei+Chen%2C+Hao+Fang%2C+Tsung%2DYi+Lin%2C+Ramakrishna+Vedantam%2C+Saurabh+Gupta%2C+Piotr+Dolla%CC%81r%2C+and+C+Lawrence+Zitnick%2E+2015%2E+Microsoft+coco+captions%3A+Data+collection+and+evaluation+server%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/)
8. Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. 2018. Discriminability objective for training descriptive captions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Ruotian+Luo%2C+Brian+Price%2C+Scott+Cohen%2C+and+Gregory+Shakhnarovich%2E+2018%2E+Discriminability+objective+for+training+descriptive+captions%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Patte)
9. Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate image captioning. In CVPR. [find](https://scholar.google.com/scholar?q=Yin+Cui%2C+Guandao+Yang%2C+Andreas+Veit%2C+Xun+Huang%2C+and+Serge+Belongie%2E+2018%2E+Learning+to+evaluate+image+captioning%2E+In+CVPR%2E)
10. Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind peoples experiences with computer-generated captions of social media images. In Proceedings of the 2017 SIGCHI Conference on Human Factors in Computing Systems. [find](https://scholar.google.com/scholar?q=Haley+MacLeod%2C+Cynthia+L%2E+Bennett%2C+Meredith+Ringel+Morris%2C+and+Edward+Cutrell%2E+2017%2E+Understanding+blind+peoples+experiences+with+computer%2Dgenerated+captions+of+social+media+images%2E+In+Proceedings+of+)
11. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition… [find](https://scholar.google.com/scholar?q=Jeffrey+Donahue%2C+Lisa+Anne+Hendricks%2C+Sergio+Guadarrama%2C+Marcus+Rohrbach%2C+Subhashini+Venugopalan%2C+Kate+Saenko%2C+and+Trevor+Darrell%2E+2015%2E+Long%2Dterm+recurrent+convolutional+networks+for+visual+recogniti) [our card](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/)
12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+770%E2%80%93+)
13. Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. 2016. Deep compositional captioning: Describing novel object categories without paired training data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages… [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Subhashini+Venugopalan%2C+Marcus+Rohrbach%2C+Raymond+Mooney%2C+Kate+Saenko%2C+and+Trevor+Darrell%2E+2016%2E+Deep+compositional+captioning%3A+Describing+novel+object+categories+without+paired+tr)
14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
15. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137. [find](https://scholar.google.com/scholar?q=Andrej+Karpathy+and+Li+Fei%2DFei%2E+2015%2E+Deep+visual%2Dsemantic+alignments+for+generating+image+descriptions%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+3128%E2%80%9331) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
16. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2016. Re-evaluating automatic Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision a… [find](https://scholar.google.com/scholar?q=Mert+Kilickaya%2C+Aykut+Erdem%2C+Nazli+Ikizler%2DCinbis%2C+and+Erkut+Erdem%2E+2016%2E+Re%2Devaluating+automatic+Steven+J+Rennie%2C+Etienne+Marcheret%2C+Youssef+Mroueh%2C+Jarret+Ross%2C+and+Vaibhava+Goel%2E+2017%2E+Self%2Dcritica) [our card](https://md.research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/)
17. Anna Rohrbach, Makarand Tapaswi, Atousa Torabi, Tegan Maharaj, Marcus Rohrbach, Sanja Fidler Christopher Pal, and Bernt Schiele. 2017a. The Joint Video and Language Understanding Workshop: MovieQA and The Large Scale Movie Description Challenge (LSMDC). [https://sites.google.com/site/](https://sites.google.com/site/) describingmovie… [↗](https://sites.google.com/site/)
18. Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017b. Movie description. International Journal of Computer Vision, 123(1):94–120. [find](https://scholar.google.com/scholar?q=Anna+Rohrbach%2C+Atousa+Torabi%2C+Marcus+Rohrbach%2C+Niket+Tandon%2C+Christopher+Pal%2C+Hugo+Larochelle%2C+Aaron+Courville%2C+and+Bernt+Schiele%2E+2017b%2E+Movie+description%2E+International+Journal+of+Computer+Vision%2C+1)
19. Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. 2017. Speaking the same language: Matching machine to human captions by adversarial training. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). [find](https://scholar.google.com/scholar?q=Rakshith+Shetty%2C+Marcus+Rohrbach%2C+Lisa+Anne+Hendricks%2C+Mario+Fritz%2C+and+Bernt+Schiele%2E+2017%2E+Speaking+the+same+language%3A+Matching+machine+to+human+captions+by+adversarial+training%2E+In+Proceedings+of+t)
20. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566–4575. [find](https://scholar.google.com/scholar?q=Ramakrishna+Vedantam%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+Cider%3A+Consensus%2Dbased+image+description+evaluation%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+) [our card](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/)
21. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057. [find](https://scholar.google.com/scholar?q=Kelvin+Xu%2C+Jimmy+Ba%2C+Ryan+Kiros%2C+Kyunghyun+Cho%2C+Aaron+Courville%2C+Ruslan+Salakhudinov%2C+Rich+Zemel%2C+and+Yoshua+Bengio%2E+2015%2E+Show%2C+attend+and+tell%3A+Neural+image+caption+generation+with+visual+attention%2E) [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)