---
title: "Self-critical Sequence Training for Image — Rennie 2016"
description: "Image captioning models are usually trained with word-level cross-entropy and teacher forcing."
authors: ["Steven J. Rennie", "Etienne Marcheret", "Youssef Mroueh", "Jarret Ross", "Vaibhava Goel"]
year: 2016
venue: "arXiv:1612.00563"
source: "https://arxiv.org/abs/1612.00563"
doi: "10.48550/arxiv.1612.00563"
arxiv: "1612.00563"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Self-critical Sequence Training for Image Captioning

# Self-critical Sequence Training for Image Captioning

**Authors:** Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, Vaibhava Goel · **Year:** 2016 · **Venue:** arXiv:1612.00563 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1612.00563) · [PDF](https://arxiv.org/pdf/1612.00563) · [DOI 10.48550/arxiv.1612.00563](https://doi.org/10.48550/arxiv.1612.00563) · [arXiv 1612.00563](https://arxiv.org/abs/1612.00563) · [OpenAlex](https://openalex.org/W2560313346)

**Citations:** 2112 citations (openalex-snapshot · 2026-10-14) · [255](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Image captioning models are usually trained with word-level cross-entropy and teacher forcing. They are then judged with non-differentiable sentence metrics such as CIDEr or BLEU. This creates two problems: exposure bias, and a gap between the training objective and the evaluation metric. Rennie et al. (IBM Watson) address both with self-critical sequence training (SCST), a REINFORCE variant. Instead of learning a separate baseline or critic network, SCST uses the score of the model's own greedy test-time output as the reference reward. A sampled caption is reinforced only when it beats the greedy caption. This lowers gradient variance, needs no extra network, and aligns training with how the model is actually decoded. The authors apply SCST to CNN+LSTM captioners, both plain and attention-based, on MSCOCO, optimising CIDEr directly. Their ensembles set a new state of the art on the official MSCOCO server. The method is simple, cheap and metric-agnostic, and it became a standard fine-tuning step for sequence generation.

## Key points
- **Model variants:** FC-2K and FC-15K (non-attention) and two spatial-attention LSTMs, Att2in and Att2all, all with 512-dimensional hidden states and embeddings. Training starts with cross-entropy (XE) and continues with SCST.
- **Greedy baseline:** the reward of the greedily decoded caption serves as the REINFORCE baseline. This avoids estimating either reward values or normalisation terms.
- **Official MSCOCO server:** a 4-model Att2all SCST ensemble raised CIDEr from the previous best of 104.9 to 114.7, with BLEU4 35.2, ROUGE-L 56.3 and METEOR 27.0.
- **Karpathy test split:** the 4-model Att2all SCST ensemble scored CIDEr 116.8 with greedy decoding and 117.5 with beam search. SCST outperformed REINFORCE and MIXER-style training of the same architecture.
- **Decoding and robustness:** after SCST, greedy decoding performs about as well as beam search. SCST-trained attention models also produced more accurate captions for unusual out-of-context images.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

255 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 238 not yet.

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://md.research.elamur.ai/free/en/papers/anderson-2017-vision-language-navigation-interpreting-visually/)
- [Findings of the Third Shared Task on Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/barrault-2018-findings-third-shared-task-multimodal/)
- [Deep Communicating Agents for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/celikyilmaz-2018-deep-communicating-agents-abstractive-summarization/)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/le-2022-coderl-mastering-code-generation-through/)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://md.research.elamur.ai/free/en/papers/lu-2022-quark-controllable-text-generation-reinforced/)
- [A Deep Reinforced Model for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://md.research.elamur.ai/free/en/papers/ramamurthy-2022-reinforcement-learning-natural-language-processing/)
- [Object Hallucination in Image Captioning](https://md.research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://md.research.elamur.ai/free/en/papers/scialom-2019-answers-unite-unsupervised-metrics-reinforced/)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://md.research.elamur.ai/free/en/papers/wang-2022-ofa-unifying-architectures-tasks-modalities/)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://md.research.elamur.ai/free/en/papers/xiao-2021-hallucination-predictive-uncertainty-conditional-language/)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/)
- [Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports](https://md.research.elamur.ai/free/en/papers/zhang-2020-optimizing-factual-correctness-summary-study/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rennie2016self,
  title = {{Self-critical Sequence Training for Image Captioning}},
  author = {Rennie, Steven J. and Marcheret, Etienne and Mroueh, Youssef and Ross, Jarret and Goel, Vaibhava},
  year = {2016},
  doi = {10.48550/arxiv.1612.00563},
  eprint = {1612.00563},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1612.00563}
}
```

- [← previous Search Space Pruning: A Simple Solution for Better Coreference Resolvers](https://md.research.elamur.ai/free/en/papers/moosavi-2016-search-space-pruning-simple-solution/)
- [next → Semantically Motivated Hebrew Verb-Noun Multi-Word Expressions Identification](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-liebeskind-semantically-motivated-hebrew-verb-noun/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. EECV, 2014. 1, 4 [find](https://scholar.google.com/scholar?q=Tsung%2DYi+Lin%2C+Michael+Maire%2C+Serge+J%2E+Belongie%2C+Lubomir+D%2E+Bourdev%2C+Ross+B%2E+Girshick%2C+James+Hays%2C+Pietro+Perona%2C+Deva+Ramanan%2C+Piotr+Dolla%CC%81r%2C+and+C%2E+Lawrence+Zitnick%2E+Microsoft+COCO%3A+common+objects+in) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
2. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP, 2014. 1 [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+C%CC%A7aglar+Gu%CC%88lc%CC%A7ehre%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+Learning+phrase+representations+using+RNN+encoder%2Ddecoder+for+statistical+machine+translatio)
3. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. CVPR, 2015. 1, 2 [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Alexander+Toshev%2C+Samy+Bengio%2C+and+Dumitru+Erhan%2E+Show+and+tell%3A+A+neural+image+caption+generator%2E+CVPR%2C+2015%2E+1%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
4. Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. CVPR, 2015. 1, 2 [find](https://scholar.google.com/scholar?q=Andrej+Karpathy+and+Fei%2DFei+Li%2E+Deep+visual%2Dsemantic+alignments+for+generating+image+descriptions%2E+CVPR%2C+2015%2E+1%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
5. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 1997. 1, 2 [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+Comput%2E%2C+1997%2E+1%2C+2)
6. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015. 1, 2, 3 [find](https://scholar.google.com/scholar?q=Kelvin+Xu%2C+Jimmy+Ba%2C+Ryan+Kiros%2C+Kyunghyun+Cho%2C+Aaron+C%2E+Courville%2C+Ruslan+Salakhutdinov%2C+Richard+S%2E+Zemel%2C+and+Yoshua+Bengio%2E+Show%2C+attend+and+tell%3A+Neural+image+caption+generation+with+visual+attent) [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
7. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. ICLR, 2015. 1, 2, 3, 4, 5, 6, 11 [find](https://scholar.google.com/scholar?q=Marc%E2%80%99Aurelio+Ranzato%2C+Sumit+Chopra%2C+Michael+Auli%2C+and+Wojciech+Zaremba%2E+Sequence+level+training+with+recurrent+neural+networks%2E+ICLR%2C+2015%2E+1%2C+2%2C+3%2C+4%2C+5%2C+6%2C+11) [our card](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
8. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS, 2015. 1, 4, 5, 11 [find](https://scholar.google.com/scholar?q=Samy+Bengio%2C+Oriol+Vinyals%2C+Navdeep+Jaitly%2C+and+Noam+Shazeer%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+In+NIPS%2C+2015%2E+1%2C+4%2C+5%2C+11)
9. Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. Neural Information Processing Systems (NIPS) 2016, 2016. 1, 4 [find](https://scholar.google.com/scholar?q=Alex+Lamb%2C+Anirudh+Goyal%2C+Ying+Zhang%2C+Saizheng+Zhang%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+Professor+forcing%3A+A+new+algorithm+for+training+recurrent+networks%2E+Neural+Information+Processing+Systems+%28NIP)
10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In ACL, 2002. 2 [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+Bleu%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL%2C+2002%2E+2) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
11. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain, 2004. 2 [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%3A+Proceedings+of+the+ACL%2D04+workshop%2C+volume+8%2E+Barcelona%2C+Spain%2C+2004%2E+2)
12. Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, volume 29, pages 65–72, 2005. 2 [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+Meteor%3A+An+automatic+metric+for+mt+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+acl+workshop+on+intrinsic+and+extrinsic+evaluatio) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
13. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, 2015. 2, 11 [find](https://scholar.google.com/scholar?q=Ramakrishna+Vedantam%2C+C%2E+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Cider%3A+Consensus%2Dbased+image+description+evaluation%2E+In+CVPR%2C+2015%2E+2%2C+11) [our card](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/)
14. Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 1998. 2, 3 [find](https://scholar.google.com/scholar?q=Richard+S%2E+Sutton+and+Andrew+G%2E+Barto%2E+Reinforcement+learning%3A+An+introduction%2E+MIT+Press%2C+1998%2E+2%2C+3)
15. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, pages 229–256, 1992. 2, 3 [find](https://scholar.google.com/scholar?q=Ronald+J%2E+Williams%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+In+Machine+Learning%2C+pages+229%E2%80%93256%2C+1992%2E+2%2C+3)
16. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. 2 [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Philipp+Moritz%2C+Sergey+Levine%2C+Michael+Jordan%2C+and+Pieter+Abbeel%2E+High%2Ddimensional+continuous+control+using+generalized+advantage+estimation%2E+arXiv+preprint+arXiv%3A1506%2E02438%2C+2015%2E+2)
17. Wojciech Zaremba and Ilya Sutskever. Reinforcement learning neural turing machines. Arxiv, 2015. 2, 3 [find](https://scholar.google.com/scholar?q=Wojciech+Zaremba+and+Ilya+Sutskever%2E+Reinforcement+learning+neural+turing+machines%2E+Arxiv%2C+2015%2E+2%2C+3)
18. Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. arXiv preprint arXiv:1402.0030, 2014. 2 — [arXiv:1402.0030](https://arxiv.org/abs/1402.0030) [find](https://scholar.google.com/scholar?q=Andriy+Mnih+and+Karol+Gregor%2E+Neural+variational+inference+and+learning+in+belief+networks%2E+arXiv+preprint+arXiv%3A1402%2E0030%2C+2014%2E+2+%E2%80%94+%5BarXiv%3A1402%2E0030%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1402%2E0030%29)
19. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. Arxiv, 2016. 2 [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Philemon+Brakel%2C+Kelvin+Xu%2C+Anirudh+Goyal%2C+Ryan+Lowe%2C+Joelle+Pineau%2C+Aaron+C%2E+Courville%2C+and+Yoshua+Bengio%2E+An+actor%2Dcritic+algorithm+for+sequence+prediction%2E+Arxiv%2C+2016%2E+2) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2016-actor-critic-algorithm-sequence-prediction/)
20. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015. 3, 5 [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+ICLR%2C+2015%2E+3%2C+5)
21. Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. 4, 6, 7 [find](https://scholar.google.com/scholar?q=Andrej+Karpathy+and+Fei%2DFei+Li%2E+Deep+visual%2Dsemantic+alignments+for+generating+image+descriptions%2E+In+CVPR%2C+2015%2E+4%2C+6%2C+7) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/)
22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 4 [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+Deep+residual+learning+for+image+recognition%2E+In+CVPR%2C+2016%2E+4)
23. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge. PAMI, 2016. 7 [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Alexander+Toshev%2C+Samy+Bengio%2C+and+Dumitru+Erhan%2E+Show+and+tell%3A+Lessons+learned+from+the+2015+MSCOCO+image+captioning+challenge%2E+PAMI%2C+2016%2E+7)
24. Myung Jinchoi, Antonio Torralba, and Alan S. Willsky. Context models and out-of-context objects. 7, 9, 11, 13, 15, 16 Self-critical Sequence Training for Image Captioning: Supplementary Material [find](https://scholar.google.com/scholar?q=Myung+Jinchoi%2C+Antonio+Torralba%2C+and+Alan+S%2E+Willsky%2E+Context+models+and+out%2Dof%2Dcontext+objects%2E+7%2C+9%2C+11%2C+13%2C+15%2C+16+Self%2Dcritical+Sequence+Training+for+Image+Captioning%3A+Supplementary+Material)