---
title: "Embedding Entities and Relations for Learning and Inference in Knowledge Bases — Yang 2014"
description: "In addition, the authors propose extracting logical Horn rules from the learned relation vectors. 35 references, cited by 458 papers here."
authors: ["Bishan Yang", "Wen-tau Yih", "Xiaodong He", "Jianfeng Gao", "Li Deng"]
year: 2014
venue: "arXiv:1412.6575"
source: "https://arxiv.org/abs/1412.6575"
doi: "10.48550/arxiv.1412.6575"
arxiv: "1412.6575"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2014-embedding-entities-relations-learning-inference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2014-embedding-entities-relations-learning-inference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/) / Embedding Entities and Relations for Learning and Inference in Knowledge Bases

# Embedding Entities and Relations for Learning and Inference in Knowledge Bases

**Authors:** [Bishan Yang](https://md.research.elamur.ai/free/en/authors/bishan-yang/), [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/), [Xiaodong He](https://md.research.elamur.ai/free/en/authors/xiaodong-he/), [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) and [Li Deng](https://md.research.elamur.ai/free/en/authors/li-deng/) · **Year:** 2014 · **Venue:** arXiv:1412.6575 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1412.6575) · [PDF](https://arxiv.org/pdf/1412.6575) · [DOI 10.48550/arxiv.1412.6575](https://doi.org/10.48550/arxiv.1412.6575) · [arXiv 1412.6575](https://arxiv.org/abs/1412.6575)

**Citations:** [458](https://md.research.elamur.ai/free/en/papers/yang-2014-embedding-entities-relations-learning-inference/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper studies how to represent entities and relations from knowledge bases (KB), such as Freebase, as compact vectors. The goal is to predict missing facts and to perform logical inference. The authors, from Cornell University and Microsoft Research, show that most well-known neural models, including NTN and TransE, are special cases of a single general framework. In it, entities are low-dimensional vectors learned by a neural network, and relations are linear and/or bilinear transformations applied to a pair of such vectors. Within this framework, the authors compare different ways of representing entities and relations on the link prediction task. The best results came from the simplest bilinear model, in which each relation is given by a diagonal matrix (DistMult). It has the same number of parameters as TransE but performs markedly better on Freebase. In addition, the authors propose extracting logical Horn rules from the learned relation vectors. Chaining two relations together is modeled as the product of their matrices. On rules involving such chains, this approach outperformed the AMIE system. The work became one of the core baselines for subsequent knowledge graph embedding models.

## Key points
- On Freebase, the proportion of correct answers among the top 10 candidates (top-10 accuracy) is 73.2% versus 54.7% for TransE. With this, the model set a new state of the art on the link prediction task.
- The common framework covers Distance, Single Layer, TransE and NTN. The most expressive of these is NTN, since it contains both a linear and a bilinear operator. TransE has the fewest parameters.
- DistMult is NTN without the nonlinear layer and the linear part, with a diagonal matrix M_r instead of a tensor. As a result, the model scales as well as TransE.
- The authors extract rules such as "BornInCity(a,b) and CityInCountry(b,c) ⇒ Nationality(a,c)". Embeddings learned with the bilinear approach capture particularly well how relations compose into chains, and on such rules the method outperforms AMIE.
- In the visualization, DistMult relation vectors form meaningful clusters: for example, /film/release_region lies close to /film/country. The vectors of the DistAdd model are much harder to interpret.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/)

## Cite

Cite as: Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao and Li Deng. 2014. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. arXiv:1412.6575.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yang2014embedding,
  title = {{Embedding Entities and Relations for Learning and Inference in Knowledge Bases}},
  author = {Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao and Li Deng},
  year = {2014},
  doi = {10.48550/arxiv.1412.6575},
  eprint = {1412.6575},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1412.6575}
}
```

- [← previous Semantic Parsing for Single-Relation Question Answering](https://md.research.elamur.ai/free/en/papers/yih-2014-semantic-parsing-single-relation-question/)
- [next → Modeling Joint Entity and Relation Extraction with Table Representation](https://md.research.elamur.ai/free/en/papers/miwa-2014-modeling-joint-entity-relation-extraction/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/)

## References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Auer, Sören, Bizer, Christian, Kobilarov, Georgi, Lehmann, Jens, Cyganiak, Richard, and Ives, Zachary. Dbpedia: A nucleus for a web of open data. In The semantic web, pp. 722–735. Springer, 2007. [find](https://scholar.google.com/scholar?q=Auer%2C+So%CC%88ren%2C+Bizer%2C+Christian%2C+Kobilarov%2C+Georgi%2C+Lehmann%2C+Jens%2C+Cyganiak%2C+Richard%2C+and+Ives%2C+Zachary%2E+Dbpedia%3A+A+nucleus+for+a+web+of+open+data%2E+In+The+semantic+web%2C+pp%2E+722%E2%80%93735%2E+Springer%2C+2007%2E)
2. Bordes, Antoine, Weston, Jason, Collobert, Ronan, and Bengio, Yoshua. Learning structured embeddings of knowledge bases. In AAAI, 2011. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Weston%2C+Jason%2C+Collobert%2C+Ronan%2C+and+Bengio%2C+Yoshua%2E+Learning+structured+embeddings+of+knowledge+bases%2E+In+AAAI%2C+2011%2E)
3. Bordes, Antoine, Glorot, Xavier, Weston, Jason, and Bengio, Yoshua. A semantic matching energy function for learning with multi-relational data. Machine Learning, pp. 1–27, 2013a. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Glorot%2C+Xavier%2C+Weston%2C+Jason%2C+and+Bengio%2C+Yoshua%2E+A+semantic+matching+energy+function+for+learning+with+multi%2Drelational+data%2E+Machine+Learning%2C+pp%2E+1%E2%80%9327%2C+2013a%2E)
4. Bordes, Antoine, Usunier, Nicolas, Garcia-Duran, Alberto, Weston, Jason, and Yakhnenko, Oksana. Translating embeddings for modeling multi-relational data. In NIPS, 2013b. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Usunier%2C+Nicolas%2C+Garcia%2DDuran%2C+Alberto%2C+Weston%2C+Jason%2C+and+Yakhnenko%2C+Oksana%2E+Translating+embeddings+for+modeling+multi%2Drelational+data%2E+In+NIPS%2C+2013b%2E)
5. Figure 3: Relation embeddings (D IST A DD ) Figure 4: Relation embeddings (D IST M ULT ) Bowman, Samuel R. Can recursive neural tensor networks learn logical reasoning? In ICLR, 2014. [find](https://scholar.google.com/scholar?q=Figure+3%3A+Relation+embeddings+%28D+IST+A+DD+%29+Figure+4%3A+Relation+embeddings+%28D+IST+M+ULT+%29+Bowman%2C+Samuel+R%2E+Can+recursive+neural+tensor+networks+learn+logical+reasoning%3F+In+ICLR%2C+2014%2E)
6. Chang, Kai-Wei, Yih, Wen-tau, Yang, Bishan, and Meek, Chris. Typed tensor decomposition of knowledge bases for relation extraction. In EMNLP, 2014. [find](https://scholar.google.com/scholar?q=Chang%2C+Kai%2DWei%2C+Yih%2C+Wen%2Dtau%2C+Yang%2C+Bishan%2C+and+Meek%2C+Chris%2E+Typed+tensor+decomposition+of+knowledge+bases+for+relation+extraction%2E+In+EMNLP%2C+2014%2E)
7. Deng, Li, Hinton, G., and Kingsbury, B. New types of deep neural network learning for speech recognition and related applications: An overview. In in ICASSP, 2013. [find](https://scholar.google.com/scholar?q=Deng%2C+Li%2C+Hinton%2C+G%2E%2C+and+Kingsbury%2C+B%2E+New+types+of+deep+neural+network+learning+for+speech+recognition+and+related+applications%3A+An+overview%2E+In+in+ICASSP%2C+2013%2E)
8. Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011. [find](https://scholar.google.com/scholar?q=Duchi%2C+John%2C+Hazan%2C+Elad%2C+and+Singer%2C+Yoram%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2121%E2%80%932159%2C+2011%2E)
9. Galárraga, Luis Antonio, Teflioudi, Christina, Hose, Katja, and Suchanek, Fabian. Amie: association rule mining under incomplete evidence in ontological knowledge bases. In WWW, 2013. [find](https://scholar.google.com/scholar?q=Gala%CC%81rraga%2C+Luis+Antonio%2C+Teflioudi%2C+Christina%2C+Hose%2C+Katja%2C+and+Suchanek%2C+Fabian%2E+Amie%3A+association+rule+mining+under+incomplete+evidence+in+ontological+knowledge+bases%2E+In+WWW%2C+2013%2E)
10. Gao, Jianfeng, Pantel, Patrick, Gamon, Michael, He, Xiaodong, Deng, Li, and Shen, Yelong. Modeling interestingness with deep neural networks. In EMNLP, 2014. [find](https://scholar.google.com/scholar?q=Gao%2C+Jianfeng%2C+Pantel%2C+Patrick%2C+Gamon%2C+Michael%2C+He%2C+Xiaodong%2C+Deng%2C+Li%2C+and+Shen%2C+Yelong%2E+Modeling+interestingness+with+deep+neural+networks%2E+In+EMNLP%2C+2014%2E)
11. Garcı́a-Durán, Alberto, Bordes, Antoine, and Usunier, Nicolas. Effective blending of two and three-way interactions for modeling multi-relational data. In Machine Learning and Knowledge Discovery in Databases, pp. 434–449. Springer, 2014. [find](https://scholar.google.com/scholar?q=Garc%C4%B1%CC%81a%2DDura%CC%81n%2C+Alberto%2C+Bordes%2C+Antoine%2C+and+Usunier%2C+Nicolas%2E+Effective+blending+of+two+and+three%2Dway+interactions+for+modeling+multi%2Drelational+data%2E+In+Machine+Learning+and+Knowledge+Discovery+in+)
12. Getoor, Lise and Taskar, Ben (eds.). Introduction to Statistical Relational Learning. The MIT Press, 2007. [find](https://scholar.google.com/scholar?q=Getoor%2C+Lise+and+Taskar%2C+Ben+%28eds%2E%29%2E+Introduction+to+Statistical+Relational+Learning%2E+The+MIT+Press%2C+2007%2E)
13. Grefenstette, Edward. Towards a formal distributional semantics: Simulating logical calculi with tensors. In *SEM, 2013. [find](https://scholar.google.com/scholar?q=Grefenstette%2C+Edward%2E+Towards+a+formal+distributional+semantics%3A+Simulating+logical+calculi+with+tensors%2E+In+%2ASEM%2C+2013%2E)
14. Hinton, Geoff, Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. Deep neural networks for acoustic modeling in speech recognition. IEEE Sig. Proc. Mag., 29:82–97, 2012. [find](https://scholar.google.com/scholar?q=Hinton%2C+Geoff%2C+Deng%2C+L%2E%2C+Yu%2C+D%2E%2C+Dahl%2C+G%2E%2C+Mohamed%2C+A%2E%2C+Jaitly%2C+N%2E%2C+Senior%2C+A%2E%2C+Vanhoucke%2C+V%2E%2C+Nguyen%2C+P%2E%2C+Sainath%2C+T%2E%2C+and+Kingsbury%2C+B%2E+Deep+neural+networks+for+acoustic+modeling+in+speech+recogniti)
15. Huang, Po-Sen, He, Xiaodong, Gao, Jianfeng, Deng, Li, Acero, Alex, and Heck, Larry. Learning deep structured semantic models for Web search using clickthrough data. In CIKM, 2013. [find](https://scholar.google.com/scholar?q=Huang%2C+Po%2DSen%2C+He%2C+Xiaodong%2C+Gao%2C+Jianfeng%2C+Deng%2C+Li%2C+Acero%2C+Alex%2C+and+Heck%2C+Larry%2E+Learning+deep+structured+semantic+models+for+Web+search+using+clickthrough+data%2E+In+CIKM%2C+2013%2E)
16. Hutchinson, B, Deng, L., and Yu, D. Tensor deep stacking networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1944–1957, 2013. [find](https://scholar.google.com/scholar?q=Hutchinson%2C+B%2C+Deng%2C+L%2E%2C+and+Yu%2C+D%2E+Tensor+deep+stacking+networks%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+35%288%29%3A1944%E2%80%931957%2C+2013%2E)
17. Jenatton, Rodolphe, Le Roux, Nicolas, Bordes, Antoine, and Obozinski, Guillaume. A latent factor model for highly multi-relational data. In NIPS, 2012. [find](https://scholar.google.com/scholar?q=Jenatton%2C+Rodolphe%2C+Le+Roux%2C+Nicolas%2C+Bordes%2C+Antoine%2C+and+Obozinski%2C+Guillaume%2E+A+latent+factor+model+for+highly+multi%2Drelational+data%2E+In+NIPS%2C+2012%2E)
18. Kemp, Charles, Tenenbaum, Joshua B, Griffiths, Thomas L, Yamada, Takeshi, and Ueda, Naonori. Learning systems of concepts with an infinite relational model. In AAAI, volume 3, pp. 5, 2006. [find](https://scholar.google.com/scholar?q=Kemp%2C+Charles%2C+Tenenbaum%2C+Joshua+B%2C+Griffiths%2C+Thomas+L%2C+Yamada%2C+Takeshi%2C+and+Ueda%2C+Naonori%2E+Learning+systems+of+concepts+with+an+infinite+relational+model%2E+In+AAAI%2C+volume+3%2C+pp%2E+5%2C+2006%2E)
19. Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. Distributed representations of words and phrases and their compositionality. In NIPS, pp. 3111–3119, 2013. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
20. Nickel, Maximilian, Tresp, Volker, and Kriegel, Hans-Peter. A three-way model for collective learning on multi-relational data. In ICML, pp. 809–816, 2011. [find](https://scholar.google.com/scholar?q=Nickel%2C+Maximilian%2C+Tresp%2C+Volker%2C+and+Kriegel%2C+Hans%2DPeter%2E+A+three%2Dway+model+for+collective+learning+on+multi%2Drelational+data%2E+In+ICML%2C+pp%2E+809%E2%80%93816%2C+2011%2E)
21. Nickel, Maximilian, Tresp, Volker, and Kriegel, Hans-Peter. Factorizing YAGO: scalable machine learning for linked data. In WWW, pp. 271–280, 2012. [find](https://scholar.google.com/scholar?q=Nickel%2C+Maximilian%2C+Tresp%2C+Volker%2C+and+Kriegel%2C+Hans%2DPeter%2E+Factorizing+YAGO%3A+scalable+machine+learning+for+linked+data%2E+In+WWW%2C+pp%2E+271%E2%80%93280%2C+2012%2E)
22. Paccanaro, Alberto and Hinton, Geoffrey E. Learning distributed representations of concepts using linear relational embedding. IEEE Transactions on Knowledge and Data Engineering, 13(2): 232–244, 2001. [find](https://scholar.google.com/scholar?q=Paccanaro%2C+Alberto+and+Hinton%2C+Geoffrey+E%2E+Learning+distributed+representations+of+concepts+using+linear+relational+embedding%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering%2C+13%282%29%3A+232%E2%80%93244%2C+2001)
23. Richardson, Matthew and Domingos, Pedro. Markov logic networks. Machine learning, 62(1-2): 107–136, 2006. [find](https://scholar.google.com/scholar?q=Richardson%2C+Matthew+and+Domingos%2C+Pedro%2E+Markov+logic+networks%2E+Machine+learning%2C+62%281%2D2%29%3A+107%E2%80%93136%2C+2006%2E)
24. Rocktäschel, Tim, Bošnjak, Matko, Singh, Sameer, and Riedel, Sebastian. Low-dimensional embeddings of logic. In ACL Workshop on Semantic Parsing, 2014. [find](https://scholar.google.com/scholar?q=Rockta%CC%88schel%2C+Tim%2C+Bos%CC%8Cnjak%2C+Matko%2C+Singh%2C+Sameer%2C+and+Riedel%2C+Sebastian%2E+Low%2Ddimensional+embeddings+of+logic%2E+In+ACL+Workshop+on+Semantic+Parsing%2C+2014%2E)
25. Schoenmackers, Stefan, Etzioni, Oren, Weld, Daniel S, and Davis, Jesse. Learning first-order horn clauses from web text. In EMNLP, 2010. [find](https://scholar.google.com/scholar?q=Schoenmackers%2C+Stefan%2C+Etzioni%2C+Oren%2C+Weld%2C+Daniel+S%2C+and+Davis%2C+Jesse%2E+Learning+first%2Dorder+horn+clauses+from+web+text%2E+In+EMNLP%2C+2010%2E)
26. Shen, Yelong, He, Xiaodong, Gao, Jianfeng, Deng, Li, and Mesnil, Gregoire. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM, 2014a. [find](https://scholar.google.com/scholar?q=Shen%2C+Yelong%2C+He%2C+Xiaodong%2C+Gao%2C+Jianfeng%2C+Deng%2C+Li%2C+and+Mesnil%2C+Gregoire%2E+A+latent+semantic+model+with+convolutional%2Dpooling+structure+for+information+retrieval%2E+In+CIKM%2C+2014a%2E)
27. Shen, Yelong, He, Xiaodong, Gao, Jianfeng, Deng, Li, and Mesnil, Grégoire. Learning semantic representations using convolutional neural networks for Web search. In WWW, pp. 373–374, 2014b. [find](https://scholar.google.com/scholar?q=Shen%2C+Yelong%2C+He%2C+Xiaodong%2C+Gao%2C+Jianfeng%2C+Deng%2C+Li%2C+and+Mesnil%2C+Gre%CC%81goire%2E+Learning+semantic+representations+using+convolutional+neural+networks+for+Web+search%2E+In+WWW%2C+pp%2E+373%E2%80%93374%2C+2014b%2E)
28. Singh, Ajit P and Gordon, Geoffrey J. Relational learning via collective matrix factorization. In KDD, pp. 650–658. ACM, 2008. [find](https://scholar.google.com/scholar?q=Singh%2C+Ajit+P+and+Gordon%2C+Geoffrey+J%2E+Relational+learning+via+collective+matrix+factorization%2E+In+KDD%2C+pp%2E+650%E2%80%93658%2E+ACM%2C+2008%2E)
29. Socher, Richard, Huval, Brody, Manning, Christopher D., and Ng, Andrew Y. Semantic compositionality through recursive matrix-vector spaces. In EMNLP-CoNLL, 2012. [our card](https://md.research.elamur.ai/free/en/papers/socher-2012-semantic-compositionality-through-recursive-matrix/)
30. Socher, Richard, Chen, Danqi, Manning, Christopher D., and Ng, Andrew Y. Reasoning with neural tensor networks for knowledge base completion. In NIPS, 2013. [our card](https://md.research.elamur.ai/free/en/papers/socher-2013-reasoning-neural-tensor-networks-knowledge/)
31. Suchanek, Fabian M, Kasneci, Gjergji, and Weikum, Gerhard. Yago: a core of semantic knowledge. In WWW, 2007. [find](https://scholar.google.com/scholar?q=Suchanek%2C+Fabian+M%2C+Kasneci%2C+Gjergji%2C+and+Weikum%2C+Gerhard%2E+Yago%3A+a+core+of+semantic+knowledge%2E+In+WWW%2C+2007%2E)
32. Sutskever, Ilya, Tenenbaum, Joshua B, and Salakhutdinov, Ruslan. Modelling relational data using Bayesian clustered tensor factorization. In NIPS, pp. 1821–1828, 2009. [find](https://scholar.google.com/scholar?q=Sutskever%2C+Ilya%2C+Tenenbaum%2C+Joshua+B%2C+and+Salakhutdinov%2C+Ruslan%2E+Modelling+relational+data+using+Bayesian+clustered+tensor+factorization%2E+In+NIPS%2C+pp%2E+1821%E2%80%931828%2C+2009%2E)
33. Vinyals, O., Jia, Y., Deng, L., and Darrell, T. Learning with recursive perceptual representations. In NIPS, 2012. [find](https://scholar.google.com/scholar?q=Vinyals%2C+O%2E%2C+Jia%2C+Y%2E%2C+Deng%2C+L%2E%2C+and+Darrell%2C+T%2E+Learning+with+recursive+perceptual+representations%2E+In+NIPS%2C+2012%2E)
34. Yih, Wen-tau, He, Xiaodong, and Meek, Christopher. Semantic parsing for single-relation question answering. In ACL, 2014. [our card](https://md.research.elamur.ai/free/en/papers/yih-2014-semantic-parsing-single-relation-question/)
35. Yu, D., Deng, L., and Seide, F. The deep tensor neural network with applications to large vocabulary speech recognition. IEEE Trans. Audio, Speech and Language Proc., 21(2):388 –396, 2013. [find](https://scholar.google.com/scholar?q=Yu%2C+D%2E%2C+Deng%2C+L%2E%2C+and+Seide%2C+F%2E+The+deep+tensor+neural+network+with+applications+to+large+vocabulary+speech+recognition%2E+IEEE+Trans%2E+Audio%2C+Speech+and+Language+Proc%2E%2C+21%282%29%3A388+%E2%80%93396%2C+2013%2E)