---
title: "Deep Captioning with Multimodal Recurrent Neural Networks"
description: "The authors propose m-RNN, a generative model that estimates the probability of each next word… 48 references, cited by 95 papers here."
authors: ["Junhua Mao", "Wei Xu", "Yi Yang", "Jiang Wang", "Zhiheng Huang", "Alan Yuille"]
year: 2014
venue: "arXiv:1412.6632 · ICLR 2015"
source: "https://arxiv.org/abs/1412.6632"
doi: "10.48550/arxiv.1412.6632"
arxiv: "1412.6632"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mao-2014-deep-captioning-multimodal-recurrent-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mao-2014-deep-captioning-multimodal-recurrent-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/) / Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

# Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

**Authors:** [Junhua Mao](https://md.research.elamur.ai/free/en/authors/junhua-mao/), [Wei Xu](https://md.research.elamur.ai/free/en/authors/wei-hong-xu/), [Yi Yang](https://md.research.elamur.ai/free/en/authors/yi-yang/), Jiang Wang, [Zhiheng Huang](https://md.research.elamur.ai/free/en/authors/zhiheng-huang/) and [Alan Yuille](https://md.research.elamur.ai/free/en/authors/alan-yuille/) · **Year:** 2014 · **Venue:** [arXiv:1412.6632 · ICLR 2015](https://md.research.elamur.ai/free/en/venues/international-conference-on-learning-representations-iclr/) ([ICLR 2014](https://md.research.elamur.ai/free/en/venues/international-conference-on-learning-representations-iclr/2014/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1412.6632) · [PDF](https://arxiv.org/pdf/1412.6632) · [DOI 10.48550/arxiv.1412.6632](https://doi.org/10.48550/arxiv.1412.6632) · [arXiv 1412.6632](https://arxiv.org/abs/1412.6632) · [OpenAlex](https://openalex.org/W1811254738)

**Citations:** 620 citations (OpenAlex, 2026-10-14) · [95](https://md.research.elamur.ai/free/en/papers/mao-2014-deep-captioning-multimodal-recurrent-neural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses automatic image captioning. Many earlier systems picked captions from a fixed sentence database and so could not describe unseen combinations of objects and scenes. The authors propose m-RNN, a generative model that estimates the probability of each next word given the preceding words and the image. Captions are produced by sampling from this distribution.

The architecture has three parts:
- a language branch with two word embedding layers and a recurrent layer;
- a deep CNN that encodes the image;
- a single multimodal layer that fuses the two.

The image features enter the multimodal layer at every time step rather than being stored in the recurrent state. The whole network is trained jointly with a log-likelihood objective, and word embeddings are learned from random initialization.

Experiments on IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO show state-of-the-art results for caption generation. The model also performs well on image-to-sentence and sentence-to-image retrieval, beating methods built specifically around ranking losses. This matters because a single probabilistic model handles both generation and retrieval.

## Key points
- Each time frame is built from five layers (a pair of word-embedding layers followed by recurrent, multimodal and softmax layers) together with a deep CNN. Weights are shared across time steps.
- Ablations over seven variants show two findings:
  - The two-layer embedding (layer II feeding the multimodal layer) beats a single 256-dimensional embedding while using fewer parameters (128·M + 128·256 vs 256·M).
  - Feeding visual features directly to the multimodal layer outperforms injecting them into the recurrent part, and the recurrent-injection variants were also harder to train.
- On IAPR TC-12 retrieval, m-RNN clearly beats bow-decaf, MLBL-F-decaf and MLBL-B-decaf, using the 100-image shortlist protocol of Kiros et al.
- BLEU (B-1 to B-4) is computed at the corpus level, following Fang et al.
- Code, refined image features and generated hypotheses are released at github.com/mjhucla/mRNN-CR. An earlier version appeared at the NIPS 2014 Deep Learning Workshop.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 84 works · 17,681 citations
- [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) — University, Los Angeles, United States · 99 works · 56,333 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang and Alan Yuille. 2014. Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN). arXiv:1412.6632 · ICLR 2015.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mao2014deep,
  title = {{Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)}},
  author = {Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang and Alan Yuille},
  year = {2014},
  doi = {10.48550/arxiv.1412.6632},
  eprint = {1412.6632},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1412.6632}
}
```

- [← previous Show and Tell: A Neural Image Caption Generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
- [next → Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality](https://md.research.elamur.ai/free/en/papers/lau-2014-machine-reading-tea-leaves-automatically/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/)

## References

48 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Barnard, Kobus, Duygulu, Pinar, Forsyth, David, De Freitas, Nando, Blei, David M, and Jordan, Michael I. Matching words and pictures. JMLR, 3:1107–1135, 2003. [find](https://scholar.google.com/scholar?q=Barnard%2C+Kobus%2C+Duygulu%2C+Pinar%2C+Forsyth%2C+David%2C+De+Freitas%2C+Nando%2C+Blei%2C+David+M%2C+and+Jordan%2C+Michael+I%2E+Matching+words+and+pictures%2E+JMLR%2C+3%3A1107%E2%80%931135%2C+2003%2E)
2. Chen, X., Fang, H., Lin, TY, Vedantam, R., Gupta, S., Dollr, P., and Zitnick, C. L. [Microsoft coco captions: Data collection and evaluation server](https://md.research.elamur.ai/free/en/papers/chen-2015-microsoft-coco-captions-data-collection/). arXiv preprint arXiv:1504.00325, 2015.
3. Chen, Xinlei and Zitnick, C Lawrence. Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654, 2014. [find](https://scholar.google.com/scholar?q=Chen%2C+Xinlei+and+Zitnick%2C+C+Lawrence%2E+Learning+a+recurrent+visual+representation+for+image+caption+generation%2E+arXiv+preprint+arXiv%3A1411%2E5654%2C+2014%2E)
4. Cho, Kyunghyun, van Merrienboer, Bart, Gulcehre, Caglar, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.[1078, 2014](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/).
5. Devlin, Jacob, Cheng, Hao, Fang, Hao, Gupta, Saurabh, Deng, Li, He, Xiaodong, Zweig, Geoffrey, and Mitchell, Margaret. Language models for image captioning: The quirks and what works. arXiv preprint arXiv:1505.01809, 2015a. [find](https://scholar.google.com/scholar?q=Devlin%2C+Jacob%2C+Cheng%2C+Hao%2C+Fang%2C+Hao%2C+Gupta%2C+Saurabh%2C+Deng%2C+Li%2C+He%2C+Xiaodong%2C+Zweig%2C+Geoffrey%2C+and+Mitchell%2C+Margaret%2E+Language+models+for+image+captioning%3A+The+quirks+and+what+works%2E+arXiv+preprint+a)
6. Devlin, Jacob, Gupta, Saurabh, Girshick, Ross, Mitchell, Margaret, and Zitnick, C Lawrence. Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467, 2015b. [find](https://scholar.google.com/scholar?q=Devlin%2C+Jacob%2C+Gupta%2C+Saurabh%2C+Girshick%2C+Ross%2C+Mitchell%2C+Margaret%2C+and+Zitnick%2C+C+Lawrence%2E+Exploring+nearest+neighbor+approaches+for+image+captioning%2E+arXiv+preprint+arXiv%3A1505%2E04467%2C+2015b%2E)
7. Donahue, Jeff, Hendricks, Lisa Anne, Guadarrama, Sergio, Rohrbach, Marcus, Venugopalan, Subhashini, Saenko, Kate, and Darrell, Trevor. [Long-term recurrent convolutional networks for visual recognition and description](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/). arXiv preprint arXiv:1411.4389, 2014.
8. Elman, Jeffrey L. Finding structure in time. Cognitive science, 14(2):179–211, 1990. [find](https://scholar.google.com/scholar?q=Elman%2C+Jeffrey+L%2E+Finding+structure+in+time%2E+Cognitive+science%2C+14%282%29%3A179%E2%80%93211%2C+1990%2E)
9. Fang, Hao, Gupta, Saurabh, Iandola, Forrest, Srivastava, Rupesh, Deng, Li, Dollár, Piotr, Gao, Jianfeng, He, Xiaodong, Mitchell, Margaret, Platt, John, et al. [From captions to visual concepts and back](https://md.research.elamur.ai/free/en/papers/fang-2014-captions-visual-concepts-back/). arXiv preprint arXiv:1411.4952, 2014.
10. Farhadi, Ali, Hejrati, Mohsen, Sadeghi, Mohammad Amin, Young, Peter, Rashtchian, Cyrus, Hockenmaier, Julia, and Forsyth, David. Every picture tells a story: Generating sentences from images. In ECCV, pp. 15–29. 2010. [find](https://scholar.google.com/scholar?q=Farhadi%2C+Ali%2C+Hejrati%2C+Mohsen%2C+Sadeghi%2C+Mohammad+Amin%2C+Young%2C+Peter%2C+Rashtchian%2C+Cyrus%2C+Hockenmaier%2C+Julia%2C+and+Forsyth%2C+David%2E+Every+picture+tells+a+story%3A+Generating+sentences+from+images%2E+In+ECCV%2C+)
11. Frome, Andrea, Corrado, Greg S, Shlens, Jon, Bengio, Samy, Dean, Jeff, Mikolov, Tomas, et al. Devise: A deep visual-semantic embedding model. In NIPS, pp. 2121–2129, 2013. [find](https://scholar.google.com/scholar?q=Frome%2C+Andrea%2C+Corrado%2C+Greg+S%2C+Shlens%2C+Jon%2C+Bengio%2C+Samy%2C+Dean%2C+Jeff%2C+Mikolov%2C+Tomas%2C+et+al%2E+Devise%3A+A+deep+visual%2Dsemantic+embedding+model%2E+In+NIPS%2C+pp%2E+2121%E2%80%932129%2C+2013%2E)
12. Girshick, R., Donahue, J., Darrell, T., and Malik, J. [Rich feature hierarchies for accurate object detection and semantic segmentation](https://md.research.elamur.ai/free/en/papers/girshick-2013-rich-feature-hierarchies-accurate-object/). In CVPR, 2014.
13. Grubinger, Michael, Clough, Paul, Müller, Henning, and Deselaers, Thomas. The iapr tc-12 benchmark: A new evaluation resource for visual information systems. In International Workshop OntoImage, pp. 13–23, 2006. [find](https://scholar.google.com/scholar?q=Grubinger%2C+Michael%2C+Clough%2C+Paul%2C+Mu%CC%88ller%2C+Henning%2C+and+Deselaers%2C+Thomas%2E+The+iapr+tc%2D12+benchmark%3A+A+new+evaluation+resource+for+visual+information+systems%2E+In+International+Workshop+OntoImage%2C+pp%2E+)
14. Guillaumin, Matthieu, Verbeek, Jakob, and Schmid, Cordelia. Multiple instance metric learning from automatically labeled bags of faces. In ECCV, pp. 634–647, 2010. [find](https://scholar.google.com/scholar?q=Guillaumin%2C+Matthieu%2C+Verbeek%2C+Jakob%2C+and+Schmid%2C+Cordelia%2E+Multiple+instance+metric+learning+from+automatically+labeled+bags+of+faces%2E+In+ECCV%2C+pp%2E+634%E2%80%93647%2C+2010%2E)
15. Gupta, Ankush and Mannem, Prashanth. From image annotation to image description. In ICONIP, 2012. [find](https://scholar.google.com/scholar?q=Gupta%2C+Ankush+and+Mannem%2C+Prashanth%2E+From+image+annotation+to+image+description%2E+In+ICONIP%2C+2012%2E)
16. Gupta, Ankush, Verma, Yashaswi, and Jawahar, CV. Choosing linguistics over vision to describe images. In AAAI, 2012. [find](https://scholar.google.com/scholar?q=Gupta%2C+Ankush%2C+Verma%2C+Yashaswi%2C+and+Jawahar%2C+CV%2E+Choosing+linguistics+over+vision+to+describe+images%2E+In+AAAI%2C+2012%2E)
17. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
18. Hodosh, Micah, Young, Peter, and Hockenmaier, Julia. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, 47:853–899, 2013. [find](https://scholar.google.com/scholar?q=Hodosh%2C+Micah%2C+Young%2C+Peter%2C+and+Hockenmaier%2C+Julia%2E+Framing+image+description+as+a+ranking+task%3A+Data%2C+models+and+evaluation+metrics%2E+JAIR%2C+47%3A853%E2%80%93899%2C+2013%2E)
19. Jia, Yangqing, Salzmann, Mathieu, and Darrell, Trevor. Learning cross-modality similarity for multinomial data. In ICCV, pp. 2407–2414, 2011. [find](https://scholar.google.com/scholar?q=Jia%2C+Yangqing%2C+Salzmann%2C+Mathieu%2C+and+Darrell%2C+Trevor%2E+Learning+cross%2Dmodality+similarity+for+multinomial+data%2E+In+ICCV%2C+pp%2E+2407%E2%80%932414%2C+2011%2E)
20. Kalchbrenner, Nal and Blunsom, Phil. [Recurrent continuous translation models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/). In EMNLP, pp. 1700–1709, 2013.
21. Karpathy, Andrej and Fei-Fei, Li. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). arXiv preprint arXiv:1412.2306, 2014.
22. Karpathy, Andrej, Joulin, Armand, and Fei-Fei, Li. Deep fragment embeddings for bidirectional image sentence mapping. In arXiv:1406.5679, 2014. [find](https://scholar.google.com/scholar?q=Karpathy%2C+Andrej%2C+Joulin%2C+Armand%2C+and+Fei%2DFei%2C+Li%2E+Deep+fragment+embeddings+for+bidirectional+image+sentence+mapping%2E+In+arXiv%3A1406%2E5679%2C+2014%2E)
23. Kiros, Ryan, Salakhutdinov, Ruslan, and Zemel, Richard S. [Unifying visual-semantic embeddings with multimodal neural language models](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/). arXiv preprint arXiv:1411.2539, 2014a.
24. Kiros, Ryan, Zemel, R, and Salakhutdinov, Ruslan. Multimodal neural language models. In ICML, 2014b. [find](https://scholar.google.com/scholar?q=Kiros%2C+Ryan%2C+Zemel%2C+R%2C+and+Salakhutdinov%2C+Ruslan%2E+Multimodal+neural+language+models%2E+In+ICML%2C+2014b%2E)
25. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS, pp. 1097–1105, 2012.
26. Kulkarni, Girish, Premraj, Visruth, Dhar, Sagnik, Li, Siming, Choi, Yejin, Berg, Alexander C, and Berg, Tamara L. Baby talk: Understanding and generating image descriptions. In CVPR, 2011. [find](https://scholar.google.com/scholar?q=Kulkarni%2C+Girish%2C+Premraj%2C+Visruth%2C+Dhar%2C+Sagnik%2C+Li%2C+Siming%2C+Choi%2C+Yejin%2C+Berg%2C+Alexander+C%2C+and+Berg%2C+Tamara+L%2E+Baby+talk%3A+Understanding+and+generating+image+descriptions%2E+In+CVPR%2C+2011%2E)
27. Kuznetsova, Polina, Ordonez, Vicente, Berg, Tamara L, and Choi, Yejin. [Treetalk: Composition and compression of trees for image descriptions](https://md.research.elamur.ai/free/en/papers/kuznetsova-2014-treetalk-composition-compression-trees-image/). Transactions of the Association for Computational Linguistics, 2(10):351–362, 2014.
28. LeCun, Yann A, Bottou, Léon, Orr, Genevieve B, and Müller, Klaus-Robert. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9–48. Springer, 2012. [find](https://scholar.google.com/scholar?q=LeCun%2C+Yann+A%2C+Bottou%2C+Le%CC%81on%2C+Orr%2C+Genevieve+B%2C+and+Mu%CC%88ller%2C+Klaus%2DRobert%2E+Efficient+backprop%2E+In+Neural+networks%3A+Tricks+of+the+trade%2C+pp%2E+9%E2%80%9348%2E+Springer%2C+2012%2E)
29. Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Dollár, Piotr, and Zitnick, C Lawrence. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). arXiv preprint arXiv:1405.0312, 2014.
30. Mao, Junhua, Xu, Wei, Yang, Yi, Wang, Jiang, and Yuille, Alan L. Explain images with multimodal recurrent neural networks. NIPS DeepLearning Workshop, 2014. [find](https://scholar.google.com/scholar?q=Mao%2C+Junhua%2C+Xu%2C+Wei%2C+Yang%2C+Yi%2C+Wang%2C+Jiang%2C+and+Yuille%2C+Alan+L%2E+Explain+images+with+multimodal+recurrent+neural+networks%2E+NIPS+DeepLearning+Workshop%2C+2014%2E)
31. Mao, Junhua, Xu, Wei, Yang, Yi, Wang, Jiang, Huang, Zhiheng, and Yuille, Alan. Learning like a child: Fast novel visual concept learning from sentence descriptions of images. arXiv preprint arXiv:1504.06692, 2015. — [arXiv:1504.06692](https://arxiv.org/abs/1504.06692) [find](https://scholar.google.com/scholar?q=Mao%2C+Junhua%2C+Xu%2C+Wei%2C+Yang%2C+Yi%2C+Wang%2C+Jiang%2C+Huang%2C+Zhiheng%2C+and+Yuille%2C+Alan%2E+Learning+like+a+child%3A+Fast+novel+visual+concept+learning+from+sentence+descriptions+of+images%2E+arXiv+preprint+arXiv%3A1504)
32. Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev. Recurrent neural network based language model. In INTERSPEECH, pp. 1045–1048, 2010. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas%2C+Karafia%CC%81t%2C+Martin%2C+Burget%2C+Lukas%2C+Cernocky%CC%80%2C+Jan%2C+and+Khudanpur%2C+Sanjeev%2E+Recurrent+neural+network+based+language+model%2E+In+INTERSPEECH%2C+pp%2E+1045%E2%80%931048%2C+2010%2E)
33. Mikolov, Tomas, Kombrink, Stefan, Burget, Lukas, Cernocky, JH, and Khudanpur, Sanjeev. Extensions of recurrent neural network language model. In ICASSP, pp. 5528–5531, 2011. [find](https://scholar.google.com/scholar?q=Mikolov%2C+Tomas%2C+Kombrink%2C+Stefan%2C+Burget%2C+Lukas%2C+Cernocky%2C+JH%2C+and+Khudanpur%2C+Sanjeev%2E+Extensions+of+recurrent+neural+network+language+model%2E+In+ICASSP%2C+pp%2E+5528%E2%80%935531%2C+2011%2E)
34. Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In NIPS, pp. 3111–3119, 2013.
35. Mitchell, Margaret, Han, Xufeng, Dodge, Jesse, Mensch, Alyssa, Goyal, Amit, Berg, Alex, Yamaguchi, Kota, Berg, Tamara, Stratos, Karl, and Daumé III, Hal. Midge: Generating image descriptions from computer vision detections. In EACL, 2012. [find](https://scholar.google.com/scholar?q=Mitchell%2C+Margaret%2C+Han%2C+Xufeng%2C+Dodge%2C+Jesse%2C+Mensch%2C+Alyssa%2C+Goyal%2C+Amit%2C+Berg%2C+Alex%2C+Yamaguchi%2C+Kota%2C+Berg%2C+Tamara%2C+Stratos%2C+Karl%2C+and+Daume%CC%81+III%2C+Hal%2E+Midge%3A+Generating+image+descriptions+from+com)
36. Mnih, Andriy and Hinton, Geoffrey. Three new graphical models for statistical language modelling. In ICML, pp. 641–648. ACM, 2007. [find](https://scholar.google.com/scholar?q=Mnih%2C+Andriy+and+Hinton%2C+Geoffrey%2E+Three+new+graphical+models+for+statistical+language+modelling%2E+In+ICML%2C+pp%2E+641%E2%80%93648%2E+ACM%2C+2007%2E)
37. Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807–814, 2010. [find](https://scholar.google.com/scholar?q=Nair%2C+Vinod+and+Hinton%2C+Geoffrey+E%2E+Rectified+linear+units+improve+restricted+boltzmann+machines%2E+In+ICML%2C+pp%2E+807%E2%80%93814%2C+2010%2E)
38. Papineni, Kishore, Roukos, Salim, Ward, Todd, and Zhu, Wei-Jing. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL, pp. 311–318, 2002.
39. Rashtchian, Cyrus, Young, Peter, Hodosh, Micah, and Hockenmaier, Julia. Collecting image annotations using amazon’s mechanical turk. In NAACL-HLT workshop 2010, pp. 139–147, 2010. — [Collecting Image Annotations Using Amazon's Mechanical Turk](https://md.research.elamur.ai/free/en/papers/rashtchian-2010-collecting-image-annotations-amazon-s/)
40. Rumelhart, David E, Hinton, Geoffrey E, and Williams, Ronald J. Learning representations by back-propagating errors. Cognitive modeling, 1988. [find](https://scholar.google.com/scholar?q=Rumelhart%2C+David+E%2C+Hinton%2C+Geoffrey+E%2C+and+Williams%2C+Ronald+J%2E+Learning+representations+by+back%2Dpropagating+errors%2E+Cognitive+modeling%2C+1988%2E)
41. Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg, Alexander C., and Fei-Fei, Li. [ImageNet Large Scale Visual Recognition Challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/), 2014.
42. Simonyan, Karen and Zisserman, Andrew. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). arXiv preprint arXiv:1409.1556, 2014.
43. Socher, Richard, Le, Q, Manning, C, and Ng, A. [Grounded compositional semantics for finding and describing images with sentences](https://md.research.elamur.ai/free/en/papers/socher-2014-grounded-compositional-semantics-finding-describing/). In TACL, 2014.
44. Srivastava, Nitish and Salakhutdinov, Ruslan. Multimodal learning with deep boltzmann machines. In NIPS, pp. 2222–2230, 2012. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Nitish+and+Salakhutdinov%2C+Ruslan%2E+Multimodal+learning+with+deep+boltzmann+machines%2E+In+NIPS%2C+pp%2E+2222%E2%80%932230%2C+2012%2E)
45. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In NIPS, pp. 3104–3112, 2014.
46. Vedantam, Ramakrishna, Zitnick, C Lawrence, and Parikh, Devi. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). arXiv preprint arXiv:1411.5726, 2014.
47. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). arXiv preprint arXiv:1411.4555, 2014.
48. Young, Peter, Lai, Alice, Hodosh, Micah, and Hockenmaier, Julia. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In ACL, pp. 479–488, 2014. [find](https://scholar.google.com/scholar?q=Young%2C+Peter%2C+Lai%2C+Alice%2C+Hodosh%2C+Micah%2C+and+Hockenmaier%2C+Julia%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+In+ACL%2C+pp%2E+479%E2%80%934)

*Updated 2 November 2026*