---
title: "SPICE: Semantic Propositional Image Caption — Anderson 2016"
description: "Automatic metrics for image captioning, such as Bleu, ROUGE, CIDEr and METEOR, mostly reward shared n-grams."
authors: ["Peter Anderson", "Basura Fernando", "Mark Johnson", "Stephen Gould"]
year: 2016
venue: "arXiv:1607.08822"
source: "https://arxiv.org/abs/1607.08822"
doi: "10.48550/arxiv.1607.08822"
arxiv: "1607.08822"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/anderson-2016-spice-semantic-propositional-image-caption/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/anderson-2016-spice-semantic-propositional-image-caption/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / SPICE: Semantic Propositional Image Caption Evaluation

# SPICE: Semantic Propositional Image Caption Evaluation

**Authors:** [Peter Anderson](https://md.research.elamur.ai/free/en/authors/peter-anderson/), [Basura Fernando](https://md.research.elamur.ai/free/en/authors/basura-fernando/), [Mark Johnson](https://md.research.elamur.ai/free/en/authors/mark-johnson/), [Stephen Gould](https://md.research.elamur.ai/free/en/authors/stephen-gould/) · **Year:** 2016 · **Venue:** arXiv:1607.08822 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1607.08822) · [PDF](https://arxiv.org/pdf/1607.08822) · [DOI 10.48550/arxiv.1607.08822](https://doi.org/10.48550/arxiv.1607.08822) · [arXiv 1607.08822](https://arxiv.org/abs/1607.08822) · [OpenAlex](https://openalex.org/W2950201573)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [244](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Automatic metrics for image captioning, such as Bleu, ROUGE, CIDEr and METEOR, mostly reward shared n-grams. This misleads them in two ways: unrelated captions that share a long phrase score highly, while paraphrases with no common words score poorly. The authors propose SPICE, which compares captions by meaning instead. Candidate and reference captions are first parsed with a dependency parser using Universal Dependency relations. A rule-based system then converts the parses into scene graphs of objects, attributes and relations. The score is an F-score over matching tuples from the candidate and reference graphs. The image itself is ignored, so evaluation is purely textual. The authors tested SPICE against human judgments on MS COCO, Flickr 8K, a composite dataset and PASCAL-50S. It tracks human ratings of machine-generated captions more closely than the n-gram metrics, especially when scores are aggregated per system. Because tuples can be split by category, SPICE can also diagnose specific skills, such as colour recognition or counting. This makes it useful both for ranking captioning models and for analysing their weaknesses.

## Key points
- On MS COCO, SPICE's system-level correlation with human judgments is 0.88, compared with 0.43 for CIDEr and 0.53 for METEOR.
- At the caption level (Kendall's τ), SPICE gains only modestly:
  - Flickr 8K: 0.45, versus 0.44 for CIDEr, 0.42 for METEOR and 0.73 for inter-human agreement.
  - Composite dataset: 0.39, versus 0.36 for CIDEr and 0.35 for METEOR.
- On PASCAL-50S with 5 references, SPICE is best at choosing between two model-generated captions (68.2% accuracy versus 66.8% for METEOR). METEOR remains best overall (80.8% versus 78.8%).
- Object counts are stored as attributes rather than duplicated nodes, so each wrong number counts as one error. An AMR representation with Smatch was tried and performed poorly.
- An analysis of 2015 COCO Captioning Challenge entries showed that the MSR system surpassed the human F-score on colour attributes, but there was little evidence that any model could count. The code is released at panderson.me/spice.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

244 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 227 not yet.

- [Explanations for CommonsenseQA: New Dataset and Models](https://md.research.elamur.ai/free/en/papers/aggarwal-2021-explanations-commonsenseqa-new-dataset-models/)
- [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://md.research.elamur.ai/free/en/papers/anderson-2017-guided-open-vocabulary-image-captioning/)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://md.research.elamur.ai/free/en/papers/chu-2023-qwen-audio-advancing-universal/)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://md.research.elamur.ai/free/en/papers/fried-2023-pragmatics-language-grounding-phenomena-tasks/)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/)
- [AudioCaps: Generating Captions for Audios in The Wild](https://md.research.elamur.ai/free/en/papers/kim-2019-audiocaps-generating-captions-audios-wild/)
- [Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness](https://md.research.elamur.ai/free/en/papers/kim-2020-will-i-sound-like-me/)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://md.research.elamur.ai/free/en/papers/lu-2022-neurologic-esque-decoding-constrained-text/)
- [Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models](https://md.research.elamur.ai/free/en/papers/oi-2026-multi-modal-multi-task-multi/)
- [Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR](https://md.research.elamur.ai/free/en/papers/opitz-2021-decomposable-metric-explainable-evaluation-text/)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://md.research.elamur.ai/free/en/papers/ramamurthy-2022-reinforcement-learning-natural-language-processing/)
- [Object Hallucination in Image Captioning](https://md.research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/)
- [Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{anderson2016spice,
  title = {{SPICE: Semantic Propositional Image Caption Evaluation}},
  author = {Anderson, Peter and Fernando, Basura and Johnson, Mark and Gould, Stephen},
  year = {2016},
  doi = {10.48550/arxiv.1607.08822},
  eprint = {1607.08822},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1607.08822}
}
```

- [← previous Speech Act Modeling of Written Asynchronous Conversations with Task-Specific Embeddings and Conditional Structured Models](https://md.research.elamur.ai/free/en/papers/joty-2016-speech-act-modeling-written-asynchronous/)
- [next → SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

45 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: CVPR. (2015) [find](https://scholar.google.com/scholar?q=Donahue%2C+J%2E%2C+Hendricks%2C+L%2EA%2E%2C+Guadarrama%2C+S%2E%2C+Rohrbach%2C+M%2E%2C+Venugopalan%2C+S%2E%2C+Saenko%2C+K%2E%2C+Darrell%2C+T%2E%3A+Long%2Dterm+recurrent+convolutional+networks+for+visual+recognition+and+description%2E+In%3A+CVPR%2E+%282015) [our card](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/)
2. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044 (2015) [find](https://scholar.google.com/scholar?q=Xu%2C+K%2E%2C+Ba%2C+J%2E%2C+Kiros%2C+R%2E%2C+Cho%2C+K%2E%2C+Courville%2C+A%2EC%2E%2C+Salakhutdinov%2C+R%2E%2C+Zemel%2C+R%2ES%2E%2C+Bengio%2C+Y%2E%3A+Show%2C+attend+and+tell%3A+Neural+image+caption+generation+with+visual+attention%2E+arXiv+preprint+arXiv%3A1502) [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
3. Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: Data, models and evaluation metrics. JAIR 47 (2013) 853–899 [find](https://scholar.google.com/scholar?q=Hodosh%2C+M%2E%2C+Young%2C+P%2E%2C+Hockenmaier%2C+J%2E%3A+Framing+image+description+as+a+ranking+task%3A+Data%2C+models+and+evaluation+metrics%2E+JAIR+47+%282013%29+853%E2%80%93899)
4. Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL 2 (2014) 67–78 [find](https://scholar.google.com/scholar?q=Young%2C+P%2E%2C+Lai%2C+A%2E%2C+Hodosh%2C+M%2E%2C+Hockenmaier%2C+J%2E%3A+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+TACL+2+%282014%29+67%E2%80%9378)
5. Lin, T., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV. (2014) [find](https://scholar.google.com/scholar?q=Lin%2C+T%2E%2C+Maire%2C+M%2E%2C+Belongie%2C+S%2E%2C+Hays%2C+J%2E%2C+Perona%2C+P%2E%2C+Ramanan%2C+D%2E%2C+Dollar%2C+P%2E%2C+Zitnick%2C+C%2EL%2E%3A+Microsoft+COCO%3A+Common+objects+in+context%2E+In%3A+ECCV%2E+%282014%29) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
6. Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollar, P., Zitnick, C.L.: Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015) [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Fang%2C+H%2E%2C+Lin%2C+T%2EY%2E%2C+Vedantam%2C+R%2E%2C+Gupta%2C+S%2E%2C+Dollar%2C+P%2E%2C+Zitnick%2C+C%2EL%2E%3A+Microsoft+COCO+captions%3A+Data+collection+and+evaluation+server%2E+arXiv+preprint+arXiv%3A1504%2E00325+%282015%29) [our card](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/)
7. Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: BabyTalk: Understanding and generating simple image descriptions. PAMI 35(12) (2013) 2891–2903 [find](https://scholar.google.com/scholar?q=Kulkarni%2C+G%2E%2C+Premraj%2C+V%2E%2C+Ordonez%2C+V%2E%2C+Dhar%2C+S%2E%2C+Li%2C+S%2E%2C+Choi%2C+Y%2E%2C+Berg%2C+A%2EC%2E%2C+Berg%2C+T%2EL%2E%3A+BabyTalk%3A+Understanding+and+generating+simple+image+descriptions%2E+PAMI+35%2812%29+%282013%29+2891%E2%80%932903)
8. Elliott, D., Keller, F.: Comparing automatic evaluation measures for image description. In: ACL. (2014) 452–457 [find](https://scholar.google.com/scholar?q=Elliott%2C+D%2E%2C+Keller%2C+F%2E%3A+Comparing+automatic+evaluation+measures+for+image+description%2E+In%3A+ACL%2E+%282014%29+452%E2%80%93457)
9. Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., Plank, B.: Automatic description generation from images: A survey of models, datasets, and evaluation measures. JAIR 55 (2016) 409–442 [find](https://scholar.google.com/scholar?q=Bernardi%2C+R%2E%2C+Cakici%2C+R%2E%2C+Elliott%2C+D%2E%2C+Erdem%2C+A%2E%2C+Erdem%2C+E%2E%2C+Ikizler%2DCinbis%2C+N%2E%2C+Keller%2C+F%2E%2C+Muscat%2C+A%2E%2C+Plank%2C+B%2E%3A+Automatic+description+generation+from+images%3A+A+survey+of+models%2C+datasets%2C+and+eval)
10. Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation. In: ACL. (2002) [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+Roukos%2C+S%2E%2C+Ward%2C+T%2E%2C+Zhu%2C+W%2E%3A+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In%3A+ACL%2E+%282002%29) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
11. Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: ACL Workshop. (2004) 25–26 [find](https://scholar.google.com/scholar?q=Lin%2C+C%2EY%2E%3A+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In%3A+ACL+Workshop%2E+%282004%29+25%E2%80%9326)
12. Vedantam, R., Zitnick, C.L., Parikh, D.: CIDEr: Consensus-based image description evaluation. In: CVPR. (2015) [find](https://scholar.google.com/scholar?q=Vedantam%2C+R%2E%2C+Zitnick%2C+C%2EL%2E%2C+Parikh%2C+D%2E%3A+CIDEr%3A+Consensus%2Dbased+image+description+evaluation%2E+In%3A+CVPR%2E+%282015%29) [our card](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/)
13. Denkowski, M., Lavie, A.: Meteor universal: Language specific translation evaluation for any target language. In: EACL 2014 Workshop on Statistical Machine Translation. (2014) [find](https://scholar.google.com/scholar?q=Denkowski%2C+M%2E%2C+Lavie%2C+A%2E%3A+Meteor+universal%3A+Language+specific+translation+evaluation+for+any+target+language%2E+In%3A+EACL+2014+Workshop+on+Statistical+Machine+Translation%2E+%282014%29) [our card](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/)
14. Giménez, J., Màrquez, L.: Linguistic features for automatic evaluation of heterogenous MT systems. In: ACL Second Workshop on Statistical Machine Translation [find](https://scholar.google.com/scholar?q=Gime%CC%81nez%2C+J%2E%2C+Ma%CC%80rquez%2C+L%2E%3A+Linguistic+features+for+automatic+evaluation+of+heterogenous+MT+systems%2E+In%3A+ACL+Second+Workshop+on+Statistical+Machine+Translation)
15. Johnson, J., Krishna, R., Stark, M., Li, L.J., Shamma, D.A., Bernstein, M.S., Fei-Fei, L.: Image retrieval using scene graphs. In: CVPR. (2015) [find](https://scholar.google.com/scholar?q=Johnson%2C+J%2E%2C+Krishna%2C+R%2E%2C+Stark%2C+M%2E%2C+Li%2C+L%2EJ%2E%2C+Shamma%2C+D%2EA%2E%2C+Bernstein%2C+M%2ES%2E%2C+Fei%2DFei%2C+L%2E%3A+Image+retrieval+using+scene+graphs%2E+In%3A+CVPR%2E+%282015%29)
16. Schuster, S., Krishna, R., Chang, A., Fei-Fei, L., Manning, C.D.: Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In: EMNLP 4th Workshop on Vision and Language. (2015) [find](https://scholar.google.com/scholar?q=Schuster%2C+S%2E%2C+Krishna%2C+R%2E%2C+Chang%2C+A%2E%2C+Fei%2DFei%2C+L%2E%2C+Manning%2C+C%2ED%2E%3A+Generating+semantically+precise+scene+graphs+from+textual+descriptions+for+improved+image+retrieval%2E+In%3A+EMNLP+4th+Workshop+on+Vision+)
17. Wang, C., Xue, N., Pradhan, S.: A transition-based algorithm for AMR parsing. In: HLT-NAACL. (2015) [find](https://scholar.google.com/scholar?q=Wang%2C+C%2E%2C+Xue%2C+N%2E%2C+Pradhan%2C+S%2E%3A+A+transition%2Dbased+algorithm+for+AMR+parsing%2E+In%3A+HLT%2DNAACL%2E+%282015%29)
18. Lin, D., Fidler, S., Kong, C., Urtasun, R.: Visual semantic search: Retrieving videos via complex textual queries. In: CVPR. (2014) [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E%2C+Fidler%2C+S%2E%2C+Kong%2C+C%2E%2C+Urtasun%2C+R%2E%3A+Visual+semantic+search%3A+Retrieving+videos+via+complex+textual+queries%2E+In%3A+CVPR%2E+%282014%29)
19. Klein, D., Manning, C.D.: Accurate unlexicalized parsing. In: ACL. (2003) [find](https://scholar.google.com/scholar?q=Klein%2C+D%2E%2C+Manning%2C+C%2ED%2E%3A+Accurate+unlexicalized+parsing%2E+In%3A+ACL%2E+%282003%29) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2003-klein-accurate-unlexicalized-parsing/)
20. De Marneffe, M.C., Dozat, T., Silveira, N., Haverinen, K., Ginter, F., Nivre, J., Manning, C.D.: Universal stanford dependencies: A cross-linguistic typology. In: LREC. Volume 14. (2014) 4585–4592 [find](https://scholar.google.com/scholar?q=De+Marneffe%2C+M%2EC%2E%2C+Dozat%2C+T%2E%2C+Silveira%2C+N%2E%2C+Haverinen%2C+K%2E%2C+Ginter%2C+F%2E%2C+Nivre%2C+J%2E%2C+Manning%2C+C%2ED%2E%3A+Universal+stanford+dependencies%3A+A+cross%2Dlinguistic+typology%2E+In%3A+LREC%2E+Volume+14%2E+%282014%29+4585%E2%80%934592) [our card](https://md.research.elamur.ai/free/en/papers/ud-2014-marneffe-universal-stanford-dependencies-cross-linguistic/)
21. Lo, C.k., Tumuluru, A.K., Wu, D.: Fully automatic semantic MT evaluation. In: ACL Seventh Workshop on Statistical Machine Translation. (2012) [find](https://scholar.google.com/scholar?q=Lo%2C+C%2Ek%2E%2C+Tumuluru%2C+A%2EK%2E%2C+Wu%2C+D%2E%3A+Fully+automatic+semantic+MT+evaluation%2E+In%3A+ACL+Seventh+Workshop+on+Statistical+Machine+Translation%2E+%282012%29)
22. Pradhan, S.S., Ward, W., Hacioglu, K., Martin, J.H., Jurafsky, D.: Shallow semantic parsing using support vector machines. In: HLT-NAACL. (2004) 233–240 [find](https://scholar.google.com/scholar?q=Pradhan%2C+S%2ES%2E%2C+Ward%2C+W%2E%2C+Hacioglu%2C+K%2E%2C+Martin%2C+J%2EH%2E%2C+Jurafsky%2C+D%2E%3A+Shallow+semantic+parsing+using+support+vector+machines%2E+In%3A+HLT%2DNAACL%2E+%282004%29+233%E2%80%93240) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2004-shallow-semantic-parsing-support-vector/)
23. Ellebracht, L., Ramisa, A., Swaroop, P., Cordero, J., Moreno-Noguer, F., Quattoni., A.: Semantic tuples for evaluation of image sentence generation. EMNLP 4th Workshop on Vision and Language (2015) [find](https://scholar.google.com/scholar?q=Ellebracht%2C+L%2E%2C+Ramisa%2C+A%2E%2C+Swaroop%2C+P%2E%2C+Cordero%2C+J%2E%2C+Moreno%2DNoguer%2C+F%2E%2C+Quattoni%2E%2C+A%2E%3A+Semantic+tuples+for+evaluation+of+image+sentence+generation%2E+EMNLP+4th+Workshop+on+Vision+and+Language+%282015%29)
24. Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K., Koehn, P., Palmer, M., Schneider, N.: Abstract meaning representation (AMR) 1.0 specification. EMNLP (2012) 1533–1544 [find](https://scholar.google.com/scholar?q=Banarescu%2C+L%2E%2C+Bonial%2C+C%2E%2C+Cai%2C+S%2E%2C+Georgescu%2C+M%2E%2C+Griffitt%2C+K%2E%2C+Hermjakob%2C+U%2E%2C+Knight%2C+K%2E%2C+Koehn%2C+P%2E%2C+Palmer%2C+M%2E%2C+Schneider%2C+N%2E%3A+Abstract+meaning+representation+%28AMR%29+1%2E0+specification%2E+EMNLP+%282012%29+)
25. Flanigan, J., Thomson, S., Carbonell, J., Dyer, C., Smith, N.A.: A discriminative graph-based parser for the abstract meaning representation. In: ACL. (2014) [find](https://scholar.google.com/scholar?q=Flanigan%2C+J%2E%2C+Thomson%2C+S%2E%2C+Carbonell%2C+J%2E%2C+Dyer%2C+C%2E%2C+Smith%2C+N%2EA%2E%3A+A+discriminative+graph%2Dbased+parser+for+the+abstract+meaning+representation%2E+In%3A+ACL%2E+%282014%29)
26. Werling, K., Angeli, G., Manning, C.: Robust subgraph generation improves abstract meaning representation parsing. In: ACL. (2015) [find](https://scholar.google.com/scholar?q=Werling%2C+K%2E%2C+Angeli%2C+G%2E%2C+Manning%2C+C%2E%3A+Robust+subgraph+generation+improves+abstract+meaning+representation+parsing%2E+In%3A+ACL%2E+%282015%29)
27. Cai, S., Knight, K.: Smatch: an evaluation metric for semantic feature structures. In: ACL (2). (2013) 748–752 [find](https://scholar.google.com/scholar?q=Cai%2C+S%2E%2C+Knight%2C+K%2E%3A+Smatch%3A+an+evaluation+metric+for+semantic+feature+structures%2E+In%3A+ACL+%282%29%2E+%282013%29+748%E2%80%93752)
28. Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: CVPR. (2015) 2641–2649 [find](https://scholar.google.com/scholar?q=Plummer%2C+B%2EA%2E%2C+Wang%2C+L%2E%2C+Cervantes%2C+C%2EM%2E%2C+Caicedo%2C+J%2EC%2E%2C+Hockenmaier%2C+J%2E%2C+Lazebnik%2C+S%2E%3A+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+models%2E+In%3A+CVPR%2E+%28)
29. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M., Fei-Fei, L.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332 (2016) [find](https://scholar.google.com/scholar?q=Krishna%2C+R%2E%2C+Zhu%2C+Y%2E%2C+Groth%2C+O%2E%2C+Johnson%2C+J%2E%2C+Hata%2C+K%2E%2C+Kravitz%2C+J%2E%2C+Chen%2C+S%2E%2C+Kalantidis%2C+Y%2E%2C+Li%2C+L%2EJ%2E%2C+Shamma%2C+D%2EA%2E%2C+Bernstein%2C+M%2E%2C+Fei%2DFei%2C+L%2E%3A+Visual+genome%3A+Connecting+language+and+vision+using+c) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
30. Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR. (June 2011) [find](https://scholar.google.com/scholar?q=Torralba%2C+A%2E%2C+Efros%2C+A%2EA%2E%3A+Unbiased+look+at+dataset+bias%2E+In%3A+CVPR%2E+%28June+2011%29)
31. Hale, J.: A probabilistic Earley parser as a psycholinguistic model. In: NAACL. (2001) 1–8 [find](https://scholar.google.com/scholar?q=Hale%2C+J%2E%3A+A+probabilistic+Earley+parser+as+a+psycholinguistic+model%2E+In%3A+NAACL%2E+%282001%29+1%E2%80%938) [our card](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/)
32. Levy, R.: Expectation-based syntactic comprehension. Cognition 106(3) (2008) 1126–1177 [find](https://scholar.google.com/scholar?q=Levy%2C+R%2E%3A+Expectation%2Dbased+syntactic+comprehension%2E+Cognition+106%283%29+%282008%29+1126%E2%80%931177)
33. Stanojević, M., Kamran, A., Koehn, P., Bojar, O.: Results of the WMT15 metrics shared task. In: ACL Tenth Workshop on Statistical Machine Translation. (2015) 256–273 [find](https://scholar.google.com/scholar?q=Stanojevic%CC%81%2C+M%2E%2C+Kamran%2C+A%2E%2C+Koehn%2C+P%2E%2C+Bojar%2C+O%2E%3A+Results+of+the+WMT15+metrics+shared+task%2E+In%3A+ACL+Tenth+Workshop+on+Statistical+Machine+Translation%2E+%282015%29+256%E2%80%93273)
34. Machacek, M., Bojar, O.: Results of the WMT14 metrics shared task. In: ACL Ninth Workshop on Statistical Machine Translation. (2014) 293–301 [find](https://scholar.google.com/scholar?q=Machacek%2C+M%2E%2C+Bojar%2C+O%2E%3A+Results+of+the+WMT14+metrics+shared+task%2E+In%3A+ACL+Ninth+Workshop+on+Statistical+Machine+Translation%2E+%282014%29+293%E2%80%93301) [our card](https://md.research.elamur.ai/free/en/papers/machacek-2014-results-wmt14-metrics-shared-task/)
35. Aditya, S., Yang, Y., Baral, C., Fermuller, C., Aloimonos, Y.: From images to sentences through scene description graphs using commonsense reasoning and knowledge. arXiv preprint arXiv:1511.03292 (2015) — [arXiv:1511.03292](https://arxiv.org/abs/1511.03292) [find](https://scholar.google.com/scholar?q=Aditya%2C+S%2E%2C+Yang%2C+Y%2E%2C+Baral%2C+C%2E%2C+Fermuller%2C+C%2E%2C+Aloimonos%2C+Y%2E%3A+From+images+to+sentences+through+scene+description+graphs+using+commonsense+reasoning+and+knowledge%2E+arXiv+preprint+arXiv%3A1511%2E03292+%28201)
36. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR. (2015) [find](https://scholar.google.com/scholar?q=Karpathy%2C+A%2E%2C+Fei%2DFei%2C+L%2E%3A+Deep+visual%2Dsemantic+alignments+for+generating+image+descriptions%2E+In%3A+CVPR%2E+%282015%29) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
37. Rashtchian, C., Young, P., Hodosh, M., Hockenmaier, J.: Collecting image annotations using Amazon’s Mechanical Turk. In: HLT-NAACL. (2010) 139–147 [find](https://scholar.google.com/scholar?q=Rashtchian%2C+C%2E%2C+Young%2C+P%2E%2C+Hodosh%2C+M%2E%2C+Hockenmaier%2C+J%2E%3A+Collecting+image+annotations+using+Amazon%E2%80%99s+Mechanical+Turk%2E+In%3A+HLT%2DNAACL%2E+%282010%29+139%E2%80%93147) [our card](https://md.research.elamur.ai/free/en/papers/rashtchian-2010-collecting-image-annotations-amazon-s/)
38. Fang, H., Gupta, S., Iandola, F.N., Srivastava, R., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J.C., Zitnick, C.L., Zweig, G.: From captions to visual concepts and back. In: CVPR. (2015) [find](https://scholar.google.com/scholar?q=Fang%2C+H%2E%2C+Gupta%2C+S%2E%2C+Iandola%2C+F%2EN%2E%2C+Srivastava%2C+R%2E%2C+Deng%2C+L%2E%2C+Dollar%2C+P%2E%2C+Gao%2C+J%2E%2C+He%2C+X%2E%2C+Mitchell%2C+M%2E%2C+Platt%2C+J%2EC%2E%2C+Zitnick%2C+C%2EL%2E%2C+Zweig%2C+G%2E%3A+From+captions+to+visual+concepts+and+back%2E+In%3A+CVPR%2E+%2820)
39. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: CVPR. (2015) [find](https://scholar.google.com/scholar?q=Vinyals%2C+O%2E%2C+Toshev%2C+A%2E%2C+Bengio%2C+S%2E%2C+Erhan%2C+D%2E%3A+Show+and+tell%3A+A+neural+image+caption+generator%2E+In%3A+CVPR%2E+%282015%29) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
40. Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., Mitchell, M.: Language models for image captioning: The quirks and what works. arXiv preprint arXiv:1505.01809 (2015) [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Cheng%2C+H%2E%2C+Fang%2C+H%2E%2C+Gupta%2C+S%2E%2C+Deng%2C+L%2E%2C+He%2C+X%2E%2C+Zweig%2C+G%2E%2C+Mitchell%2C+M%2E%3A+Language+models+for+image+captioning%3A+The+quirks+and+what+works%2E+arXiv+preprint+arXiv%3A1505%2E01809+%282015%29)
41. Mao, J., Wei, X., Yang, Y., Wang, J., Huang, Z., Yuille, A.L.: Learning like a child: Fast novel visual concept learning from sentence descriptions of images. In: CVPR. (2015) 2533–2541 [find](https://scholar.google.com/scholar?q=Mao%2C+J%2E%2C+Wei%2C+X%2E%2C+Yang%2C+Y%2E%2C+Wang%2C+J%2E%2C+Huang%2C+Z%2E%2C+Yuille%2C+A%2EL%2E%3A+Learning+like+a+child%3A+Fast+novel+visual+concept+learning+from+sentence+descriptions+of+images%2E+In%3A+CVPR%2E+%282015%29+2533%E2%80%932541)
42. Devlin, J., Gupta, S., Girshick, R.B., Mitchell, M., Zitnick, C.L.: Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467 (2015) — [arXiv:1505.04467](https://arxiv.org/abs/1505.04467) [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Gupta%2C+S%2E%2C+Girshick%2C+R%2EB%2E%2C+Mitchell%2C+M%2E%2C+Zitnick%2C+C%2EL%2E%3A+Exploring+nearest+neighbor+approaches+for+image+captioning%2E+arXiv+preprint+arXiv%3A1505%2E04467+%282015%29+%E2%80%94+%5BarXiv%3A1505%2E04467%5D%28https%3A%2F%2Farxi)
43. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A.: Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632 (2014) [find](https://scholar.google.com/scholar?q=Mao%2C+J%2E%2C+Xu%2C+W%2E%2C+Yang%2C+Y%2E%2C+Wang%2C+J%2E%2C+Huang%2C+Z%2E%2C+Yuille%2C+A%2E%3A+Deep+captioning+with+multimodal+recurrent+neural+networks+%28m%2Drnn%29%2E+arXiv+preprint+arXiv%3A1412%2E6632+%282014%29)
44. Kolár, M., Hradis, M., Zemcı́k, P.: Technical report: Image captioning with semantically similar images. arXiv preprint arXiv:1506.03995 (2015) — [arXiv:1506.03995](https://arxiv.org/abs/1506.03995) [find](https://scholar.google.com/scholar?q=Kola%CC%81r%2C+M%2E%2C+Hradis%2C+M%2E%2C+Zemc%C4%B1%CC%81k%2C+P%2E%3A+Technical+report%3A+Image+captioning+with+semantically+similar+images%2E+arXiv+preprint+arXiv%3A1506%2E03995+%282015%29+%E2%80%94+%5BarXiv%3A1506%2E03995%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1506%2E03995%29)
45. Kiros, R., Salakhutdinov, R., Zemel, R.S.: Multimodal neural language models. ICML 14 (2014) 595–603 [find](https://scholar.google.com/scholar?q=Kiros%2C+R%2E%2C+Salakhutdinov%2C+R%2E%2C+Zemel%2C+R%2ES%2E%3A+Multimodal+neural+language+models%2E+ICML+14+%282014%29+595%E2%80%93603)