---
title: "Order-Embeddings of Images and Language — Vendrov 2015"
description: "The paper proposes order-embeddings: nonnegative vectors compared under a reversed product… 35 references, cited by 106 papers here."
authors: ["Ivan Vendrov", "Ryan Kiros", "Sanja Fidler", "Raquel Urtasun"]
year: 2015
venue: "arXiv:1511.06361"
source: "https://arxiv.org/abs/1511.06361"
doi: "10.48550/arxiv.1511.06361"
arxiv: "1511.06361"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vendrov-2015-order-embeddings-images-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vendrov-2015-order-embeddings-images-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Order-Embeddings of Images and Language

# Order-Embeddings of Images and Language

**Authors:** Ivan Vendrov, [Ryan Kiros](https://md.research.elamur.ai/free/en/authors/ryan-kiros/), [Sanja Fidler](https://md.research.elamur.ai/free/en/authors/sanja-fidler/) and [Raquel Urtasun](https://md.research.elamur.ai/free/en/authors/raquel-urtasun/) · **Year:** 2015 · **Venue:** arXiv:1511.06361 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1511.06361) · [PDF](https://arxiv.org/pdf/1511.06361) · [DOI 10.48550/arxiv.1511.06361](https://doi.org/10.48550/arxiv.1511.06361) · [arXiv 1511.06361](https://arxiv.org/abs/1511.06361) · [OpenAlex](https://openalex.org/W2172888184)

**Citations:** 71 citations (OpenAlex, 2026-10-14) · [106](https://md.research.elamur.ai/free/en/papers/vendrov-2015-order-embeddings-images-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors argue that hypernymy between words, entailment between sentences, and the link between images and their captions are all instances of one partial order, which they call the visual-semantic hierarchy. In this order, general concepts sit above specific ones, and the relation is transitive and antisymmetric. Common embedding approaches compare items with symmetric measures such as cosine distance, so they cannot express which item is more general.

The paper proposes order-embeddings: nonnegative vectors compared under a reversed product order. The origin is the most general concept, and an item counts as more specific when every coordinate is at least as large. Violations of this order are penalized softly inside a max-margin loss, and the penalty can replace the comparison function in existing models.

Trained only on the WordNet hierarchy, the full model was the only variant that beat a transitive-closure baseline on withheld hypernym pairs. The authors also report better caption-image retrieval and near state-of-the-art natural language inference. The broader point is that hierarchy is built into the geometry rather than left for models to learn from data.

## Key points
- The order-violation penalty is E(x, y) = ‖max(0, y − x)‖². It equals zero exactly when the pair satisfies the reversed product order on R^N₊.
- The hypernym data is the transitive closure of WordNet: 838,073 edges among 82,192 concepts. As in Bordes et al. (2011), 4,000 edges are held out for test and another 4,000 for development.
- Training setup:
  - 50-dimensional nonnegative vectors, one per concept, stored as a lookup table.
  - Margin α = 1, with 500 true and 500 corrupted pairs per batch.
  - Adam optimizer at learning rate 0.01, 30–50 epochs, with early stopping.
- Comparisons on WordNet:
  - Baselines: transitive closure (no learning) and word2gauss (Vilnis & McCallum, 2015), which uses Gaussian embeddings with KL divergence.
  - Ablations: the same model with a symmetric cosine comparison, and with a bilinear comparison as in Socher et al. (2013).
  - Only the full asymmetric penalty surpassed the transitive-closure baseline.
- The supplementary material reports a new regularity in embeddings learned on COCO caption-image retrieval: elementwise max roughly corresponds to composition and elementwise min to abstraction. For example, max("man", "cat") retrieves images showing both.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 110 works · 179,062 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Ivan Vendrov, Ryan Kiros, Sanja Fidler and Raquel Urtasun. 2015. Order-Embeddings of Images and Language. arXiv:1511.06361.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{vendrov2015order,
  title = {{Order-Embeddings of Images and Language}},
  author = {Ivan Vendrov, Ryan Kiros, Sanja Fidler and Raquel Urtasun},
  year = {2015},
  doi = {10.48550/arxiv.1511.06361},
  eprint = {1511.06361},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06361}
}
```

- [← previous Joint Lemmatization and Morphological Tagging with Lemming](https://md.research.elamur.ai/free/en/papers/muller-2015-joint-lemmatization-morphological-tagging-lemming/)
- [next → It Depends: Dependency Parser Comparison Using A Web-based Evaluation Tool](https://md.research.elamur.ai/free/en/papers/choi-2015-it-depends-dependency-parser-comparison/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baroni, Marco, Bernardi, Raffaella, Do, Ngoc-Quynh, and Shan, Chung-chieh. [Entailment above the word level in distributional semantics](https://md.research.elamur.ai/free/en/papers/baroni-2012-entailment-above-word-level-distributional/). In EACL, 2012.
2. Bordes, Antoine, Weston, Jason, Collobert, Ronan, and Bengio, Yoshua. Learning structured embeddings of knowledge bases. In AAAI, 2011. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Weston%2C+Jason%2C+Collobert%2C+Ronan%2C+and+Bengio%2C+Yoshua%2E+Learning+structured+embeddings+of+knowledge+bases%2E+In+AAAI%2C+2011%2E)
3. Bowman, Samuel R., Angeli, Gabor, Potts, Christopher, and Manning, Christopher D. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In EMNLP, 2015.
4. Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. — [Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/)
5. Chopra, Sumit, Hadsell, Raia, and LeCun, Yann. Learning a similarity metric discriminatively, with application to face verification. In CVPR, 2005. [find](https://scholar.google.com/scholar?q=Chopra%2C+Sumit%2C+Hadsell%2C+Raia%2C+and+LeCun%2C+Yann%2E+Learning+a+similarity+metric+discriminatively%2C+with+application+to+face+verification%2E+In+CVPR%2C+2005%2E)
6. Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. [find](https://scholar.google.com/scholar?q=Deng%2C+Jia%2C+Dong%2C+Wei%2C+Socher%2C+Richard%2C+Li%2C+Li%2DJia%2C+Li%2C+Kai%2C+and+Fei%2DFei%2C+Li%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+CVPR%2C+2009%2E)
7. Frome, Andrea, Corrado, Greg S, Shlens, Jon, Bengio, Samy, Dean, Jeff, Mikolov, Tomas, et al. Devise: A deep visual-semantic embedding model. In NIPS, 2013. [find](https://scholar.google.com/scholar?q=Frome%2C+Andrea%2C+Corrado%2C+Greg+S%2C+Shlens%2C+Jon%2C+Bengio%2C+Samy%2C+Dean%2C+Jeff%2C+Mikolov%2C+Tomas%2C+et+al%2E+Devise%3A+A+deep+visual%2Dsemantic+embedding+model%2E+In+NIPS%2C+2013%2E)
8. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. ICCV, 2015. [find](https://scholar.google.com/scholar?q=He%2C+Kaiming%2C+Zhang%2C+Xiangyu%2C+Ren%2C+Shaoqing%2C+and+Sun%2C+Jian%2E+Delving+deep+into+rectifiers%3A+Surpassing+human%2Dlevel+performance+on+imagenet+classification%2E+ICCV%2C+2015%2E)
9. Hodosh, Micah, Young, Peter, and Hockenmaier, Julia. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR, 2013. [find](https://scholar.google.com/scholar?q=Hodosh%2C+Micah%2C+Young%2C+Peter%2C+and+Hockenmaier%2C+Julia%2E+Framing+image+description+as+a+ranking+task%3A+Data%2C+models+and+evaluation+metrics%2E+JAIR%2C+2013%2E)
10. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015. [find](https://scholar.google.com/scholar?q=Ioffe%2C+Sergey+and+Szegedy%2C+Christian%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+ICML%2C+2015%2E)
11. Karpathy, Andrej and Li, Fei-Fei. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). In CVPR, 2015.
12. Kingma, Diederik and Ba, Jimmy. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In ICLR, 2015.
13. Kiros, Ryan, Salakhutdinov, Ruslan, and Zemel, Richard S. [Unifying visual-semantic embeddings with multimodal neural language models](https://md.research.elamur.ai/free/en/papers/kiros-2014-unifying-visual-semantic-embeddings-multimodal/). arXiv preprint arXiv:1411.2539, 2014.
14. Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan, Zemel, Richard S, Torralba, Antonio, Urtasun, Raquel, and Fidler, Sanja. Skip-thought vectors. NIPS, 2015. [find](https://scholar.google.com/scholar?q=Kiros%2C+Ryan%2C+Zhu%2C+Yukun%2C+Salakhutdinov%2C+Ruslan%2C+Zemel%2C+Richard+S%2C+Torralba%2C+Antonio%2C+Urtasun%2C+Raquel%2C+and+Fidler%2C+Sanja%2E+Skip%2Dthought+vectors%2E+NIPS%2C+2015%2E)
15. Klein, Benjamin, Lev, Guy, Sadeh, Gil, and Wolf, Lior. Associating neural word embeddings with deep image representations using fisher vectors. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Klein%2C+Benjamin%2C+Lev%2C+Guy%2C+Sadeh%2C+Gil%2C+and+Wolf%2C+Lior%2E+Associating+neural+word+embeddings+with+deep+image+representations+using+fisher+vectors%2E+In+CVPR%2C+2015%2E)
16. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS, 2012.
17. Lin, Dahua, Fidler, Sanja, Kong, Chen, and Urtasun, Raquel. Visual semantic search: Retrieving videos via complex textual queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014a. [find](https://scholar.google.com/scholar?q=Lin%2C+Dahua%2C+Fidler%2C+Sanja%2C+Kong%2C+Chen%2C+and+Urtasun%2C+Raquel%2E+Visual+semantic+search%3A+Retrieving+videos+via+complex+textual+queries%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+)
18. Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Dollár, Piotr, and Zitnick, C Lawrence. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In ECCV, 2014b.
19. Ma, Lin, Lu, Zhengdong, Shang, Lifeng, and Li, Hang. Multimodal convolutional neural networks for matching image and sentence. ICCV, 2015. [find](https://scholar.google.com/scholar?q=Ma%2C+Lin%2C+Lu%2C+Zhengdong%2C+Shang%2C+Lifeng%2C+and+Li%2C+Hang%2E+Multimodal+convolutional+neural+networks+for+matching+image+and+sentence%2E+ICCV%2C+2015%2E)
20. Mao, Junhua, Xu, Wei, Yang, Yi, Wang, Jiang, and Yuille, Alan. [Deep captioning with multimodal recurrent neural networks (m-rnn)](https://md.research.elamur.ai/free/en/papers/mao-2014-deep-captioning-multimodal-recurrent-neural/). In ICLR, 2015.
21. Mikolov, Tomas, Yih, Wen-tau, and Zweig, Geoffrey. [Linguistic regularities in continuous space word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/). In HLT-NAACL, pp. 746–751, 2013.
22. Miller, George A. [Wordnet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM, 1995.
23. Norouzi, Mohammad, Mikolov, Tomas, Bengio, Samy, Singer, Yoram, Shlens, Jonathon, Frome, Andrea, Corrado, Greg S, and Dean, Jeffrey. Zero-shot learning by convex combination of semantic embeddings. In ICLR, 2014. [find](https://scholar.google.com/scholar?q=Norouzi%2C+Mohammad%2C+Mikolov%2C+Tomas%2C+Bengio%2C+Samy%2C+Singer%2C+Yoram%2C+Shlens%2C+Jonathon%2C+Frome%2C+Andrea%2C+Corrado%2C+Greg+S%2C+and+Dean%2C+Jeffrey%2E+Zero%2Dshot+learning+by+convex+combination+of+semantic+embeddings%2E+In)
24. Plummer, Bryan, Wang, Liwei, Cervantes, Chris, Caicedo, Juan, Hockenmaier, Julia, and Lazebnik, Svetlana. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. arXiv preprint arXiv:1505.04870, 2015. [find](https://scholar.google.com/scholar?q=Plummer%2C+Bryan%2C+Wang%2C+Liwei%2C+Cervantes%2C+Chris%2C+Caicedo%2C+Juan%2C+Hockenmaier%2C+Julia%2C+and+Lazebnik%2C+Svetlana%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsentence+m)
25. Rocktäschel, Tim, Grefenstette, Edward, Hermann, Karl Moritz, Kočiskỳ, Tomáš, and Blunsom, Phil. [Reasoning about entailment with neural attention](https://md.research.elamur.ai/free/en/papers/rocktaschel-2015-reasoning-about-entailment-neural-attention/). arXiv preprint arXiv:1509.06664, 2015.
26. Simonyan, K. and Zisserman, A. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). In ICLR, 2015.
27. Socher, Richard, Chen, Danqi, Manning, Christopher D, and Ng, Andrew. [Reasoning with neural tensor networks for knowledge base completion](https://md.research.elamur.ai/free/en/papers/socher-2013-reasoning-neural-tensor-networks-knowledge/). In NIPS, 2013.
28. Socher, Richard, Karpathy, Andrej, Le, Quoc V, Manning, Christopher D, and Ng, Andrew Y. [Grounded compositional semantics for finding and describing images with sentences](https://md.research.elamur.ai/free/en/papers/socher-2014-grounded-compositional-semantics-finding-describing/). TACL, 2014.
29. Van der Maaten, Laurens and Hinton, Geoffrey. Visualizing data using t-sne. Journal of Machine Learning Research, 9(2579-2605):85, 2008. [find](https://scholar.google.com/scholar?q=Van+der+Maaten%2C+Laurens+and+Hinton%2C+Geoffrey%2E+Visualizing+data+using+t%2Dsne%2E+Journal+of+Machine+Learning+Research%2C+9%282579%2D2605%29%3A85%2C+2008%2E)
30. Vilnis, Luke and McCallum, Andrew. Word representations via gaussian embedding. In ICLR, 2015. [find](https://scholar.google.com/scholar?q=Vilnis%2C+Luke+and+McCallum%2C+Andrew%2E+Word+representations+via+gaussian+embedding%2E+In+ICLR%2C+2015%2E)
31. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). In CVPR, 2015.
32. Young, Peter, Lai, Alice, Hodosh, Micah, and Hockenmaier, Julia. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2014. [find](https://scholar.google.com/scholar?q=Young%2C+Peter%2C+Lai%2C+Alice%2C+Hodosh%2C+Micah%2C+and+Hockenmaier%2C+Julia%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+TACL%2C+2014%2E)
33. S UPPLEMENTARY M ATERIAL Mikolov et al. (2013) showed that word representations learned using word2vec exhibit semantic regularities, such as king − man + woman ∼ queen. [find](https://scholar.google.com/scholar?q=S+UPPLEMENTARY+M+ATERIAL+Mikolov+et+al%2E+%282013%29+showed+that+word+representations+learned+using+word2vec+exhibit+semantic+regularities%2C+such+as+king+%E2%88%92+man+%2B+woman+%E2%88%BC+queen%2E)
34. Kiros et al. (2014) showed that similar regularities hold for joint image-language models. We find that order-embeddings exhibit a novel form of regularity, shown in Figure 4. The elementwise max and min operations in the embedding space roughly correspond to composition and abstraction, respectivel… [find](https://scholar.google.com/scholar?q=Kiros+et+al%2E+%282014%29+showed+that+similar+regularities+hold+for+joint+image%2Dlanguage+models%2E+We+find+that+order%2Dembeddings+exhibit+a+novel+form+of+regularity%2C+shown+in+Figure+4%2E+The+elementwise+max+and+)
35. Query max(“man”, “cat”) max(“black dog”, “park”) , ( ) max , ( ) min , ( ) “dog” min , ( ) “man” max Figure 4: Multimodal regularities found with embeddings learned for the caption-image retrieval task. Note that some images have been slightly cropped for easier viewing, but no relevant objects have… [find](https://scholar.google.com/scholar?q=Query+max%28%E2%80%9Cman%E2%80%9D%2C+%E2%80%9Ccat%E2%80%9D%29+max%28%E2%80%9Cblack+dog%E2%80%9D%2C+%E2%80%9Cpark%E2%80%9D%29+%2C+%28+%29+max+%2C+%28+%29+min+%2C+%28+%29+%E2%80%9Cdog%E2%80%9D+min+%2C+%28+%29+%E2%80%9Cman%E2%80%9D+max+Figure+4%3A+Multimodal+regularities+found+with+embeddings+learned+for+the+caption%2Dimage+retrieval+tas)

*Updated 2 November 2026*