---
title: "Observed versus latent features for knowledge base and text inference — Toutanova 2015"
description: "The authors compare two approaches. 21 references, cited by 210 papers here."
authors: ["Kristina Toutanova", "Danqi Chen"]
year: 2015
venue: "Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality"
source: "https://aclanthology.org/W15-4007/"
doi: "10.18653/v1/w15-4007"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/toutanova-2015-observed-versus-latent-features-knowledge/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/toutanova-2015-observed-versus-latent-features-knowledge/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Observed versus latent features for knowledge base and text inference

# Observed versus latent features for knowledge base and text inference

**Authors:** [Kristina Toutanova](https://md.research.elamur.ai/free/en/authors/kristina-toutanova/) and [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2015 · **Venue:** Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W15-4007/) · [PDF](https://aclanthology.org/W15-4007.pdf) · [DOI 10.18653/v1/w15-4007](https://doi.org/10.18653/v1/w15-4007)

**Citations:** [210](https://md.research.elamur.ai/free/en/papers/toutanova-2015-observed-versus-latent-features-knowledge/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper addresses automatic knowledge base completion: given known "entity–relation–entity" triples, a model must find missing links. The authors compare two approaches. The first is latent feature models, i.e., vector representations of entities and relations (model E, DistMult). The second is a very simple observed feature model. It looks only at direct links between a pair of candidates and at how often a particular entity occupies a given position in a relation. On the popular benchmarks FB15K and WN18, this simple model clearly outperforms the latent ones. The reason lies in the redundancy of these graphs: they contain many near-duplicate and inverse relations. The authors therefore created a more challenging dataset, FB15KSelected, from which such relations were removed, and on it the latent models are stronger. They then added textual relations from the ClueWeb12 corpus to the graph. The best result was achieved by a joint model combining both types of features. The paper concludes that standard benchmarks can yield inflated scores and that web text substantially helps in completing knowledge bases.

## Key points
- Latent models: model E (similar to the universal schema approach) and DistMult (a diagonal bilinear variant of RESCAL). Observed features: a simplified analogue of PRA that considers only paths of length one. All models share the same loss function: negative log-likelihood with L2 regularization.
- On FB15KSelected without text, the best observed feature model achieves an MRR of 23.5, while the E+DistMult combination achieves 26.6.
- Textual relations were extracted from 200 million parsed ClueWeb12 sentences with Freebase annotations (FACC1). Each relation is represented by a lexicalized dependency path. After filtering, 25,000 textual relations remained, yielding 6.6 million new edges in the graph. Textual mentions exist for only 23.3% of test entity pairs.
- If textual and KB relations are weighted equally in the loss function (τ = 1), there is no benefit. Reducing the weight of text to τ = 0.1 raises the MRR of the E+DistMult model to 27.4.
- The observed feature model with text increases MRR on triples with mentions from 19.3 to 39.6. Combining latent and observed features yields an overall MRR of 29.3 and HITS@10 of 46.2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

💡 Concepts (1): [🧠 Knowledge representation and reasoning](https://md.research.elamur.ai/free/en/concepts/knowledge-representation/) · 30 works

📚 Shelves (1): [🤖 AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) · 347 works

🧭 Topics (1): [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) · 87 works

## Cite

Cite as: Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{toutanova2015observed,
  title = {{Observed versus latent features for knowledge base and text inference}},
  author = {Kristina Toutanova and Danqi Chen},
  booktitle = {Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality},
  year = {2015},
  doi = {10.18653/v1/w15-4007},
  url = {https://doi.org/10.18653/v1/w15-4007}
}
```

- [← previous Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking](https://md.research.elamur.ai/free/en/papers/wen-2015-stochastic-language-generation-dialogue-recurrent/)
- [next → The CoNLL-2015 Shared Task on Shallow Discourse Parsing](https://md.research.elamur.ai/free/en/papers/xue-2015-conll-2015-shared-task-shallow/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. Springer. [find](https://scholar.google.com/scholar?q=So%CC%88ren+Auer%2C+Christian+Bizer%2C+Georgi+Kobilarov%2C+Jens+Lehmann%2C+Richard+Cyganiak%2C+and+Zachary+Ives%2E+2007%2E+Dbpedia%3A+A+nucleus+for+a+web+of+open+data%2E+Springer%2E)
2. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250. ACM. [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+a+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+the+2008+ACM+SI)
3. Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embeddings of knowledge bases. In Conference on Artificial Intelligence, number EPFL-CONF-192344. [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Jason+Weston%2C+Ronan+Collobert%2C+and+Yoshua+Bengio%2E+2011%2E+Learning+structured+embeddings+of+knowledge+bases%2E+In+Conference+on+Artificial+Intelligence%2C+number+EPFL%2DCONF%2D192344%2E)
4. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Nicolas+Usunier%2C+Alberto+Garcia%2DDuran%2C+Jason+Weston%2C+and+Oksana+Yakhnenko%2E+2013%2E+Translating+embeddings+for+modeling+multi%2Drelational+data%2E+In+Advances+in+Neural+Information+Processing)
5. Kai-Wei Chang, Wen-tau Yih, Bishan Yang, and Christopher Meek. 2014. Typed tensor decomposition of knowledge bases for relation extraction. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Kai%2DWei+Chang%2C+Wen%2Dtau+Yih%2C+Bishan+Yang%2C+and+Christopher+Meek%2E+2014%2E+Typed+tensor+decomposition+of+knowledge+bases+for+relation+extraction%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
6. Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In International Conference on Knowledge Discovery and Data Mining (KDD). [find](https://scholar.google.com/scholar?q=Xin+Dong%2C+Evgeniy+Gabrilovich%2C+Geremy+Heitz%2C+Wilko+Horn%2C+Ni+Lao%2C+Kevin+Murphy%2C+Thomas+Strohmann%2C+Shaohua+Sun%2C+and+Wei+Zhang%2E+2014%2E+Knowledge+vault%3A+A+web%2Dscale+approach+to+probabilistic+knowledge+fusi)
7. Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013. FACC1: Freebase annotation of ClueWeb corpora, Version 1 (release date 2013-06-26, format version 1, correction level 0). [find](https://scholar.google.com/scholar?q=Evgeniy+Gabrilovich%2C+Michael+Ringgaard%2C+and+Amarnag+Subramanya%2E+2013%2E+FACC1%3A+Freebase+annotation+of+ClueWeb+corpora%2C+Version+1+%28release+date+2013%2D06%2D26%2C+format+version+1%2C+correction+level+0%29%2E)
8. Matt Gardner, Partha Pratim Talukdar, Bryan Kisiel, and Tom Mitchell. 2013. Improving learning and inference in a large knowledge-base using latent syntactic cues. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Partha+Pratim+Talukdar%2C+Bryan+Kisiel%2C+and+Tom+Mitchell%2E+2013%2E+Improving+learning+and+inference+in+a+large+knowledge%2Dbase+using+latent+syntactic+cues%2E+In+Empirical+Methods+in+Natural+Lang)
9. Matt Gardner, Partha Talukdar, Jayant Krishnamurthy, and Tom Mitchell. 2014. Incorporating vector space similarity in random walk inference over knowledge bases. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Partha+Talukdar%2C+Jayant+Krishnamurthy%2C+and+Tom+Mitchell%2E+2014%2E+Incorporating+vector+space+similarity+in+random+walk+inference+over+knowledge+bases%2E+In+Empirical+Methods+in+Natural+Langua)
10. Ni Lao and William W Cohen. 2010. Relational retrieval using a combination of path-constrained random walks. Machine learning. [find](https://scholar.google.com/scholar?q=Ni+Lao+and+William+W+Cohen%2E+2010%2E+Relational+retrieval+using+a+combination+of+path%2Dconstrained+random+walks%2E+Machine+learning%2E)
11. Ni Lao, Tom Mitchell, and William W Cohen. 2011. Random walk inference and learning in a large scale knowledge base. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Ni+Lao%2C+Tom+Mitchell%2C+and+William+W+Cohen%2E+2011%2E+Random+walk+inference+and+learning+in+a+large+scale+knowledge+base%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
12. Ni Lao, Amarnag Subramanya, Fernando Pereira, and William W Cohen. 2012. Reading the web with learned syntactic-semantic inference rules. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL). [find](https://scholar.google.com/scholar?q=Ni+Lao%2C+Amarnag+Subramanya%2C+Fernando+Pereira%2C+and+William+W+Cohen%2E+2012%2E+Reading+the+web+with+learned+syntactic%2Dsemantic+inference+rules%2E+In+Empirical+Methods+in+Natural+Language+Processing+and+Comput)
13. Dong C Liu and Jorge Nocedal. 1989. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3):503–528. [find](https://scholar.google.com/scholar?q=Dong+C+Liu+and+Jorge+Nocedal%2E+1989%2E+On+the+limited+memory+bfgs+method+for+large+scale+optimization%2E+Mathematical+programming%2C+45%281%2D3%29%3A503%E2%80%93528%2E)
14. Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 809–816. [find](https://scholar.google.com/scholar?q=Maximilian+Nickel%2C+Volker+Tresp%2C+and+Hans%2DPeter+Kriegel%2E+2011%2E+A+three%2Dway+model+for+collective+learning+on+multi%2Drelational+data%2E+In+Proceedings+of+the+28th+international+conference+on+machine+learni)
15. Maximilian Nickel, Xueyan Jiang, and Volker Tresp. 2014. Reducing the rank in relational factorization models by including observable patterns. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 1179–1187.… [find](https://scholar.google.com/scholar?q=Maximilian+Nickel%2C+Xueyan+Jiang%2C+and+Volker+Tresp%2E+2014%2E+Reducing+the+rank+in+relational+factorization+models+by+including+observable+patterns%2E+In+Z%2E+Ghahramani%2C+M%2E+Welling%2C+C%2E+Cortes%2C+N%2ED%2E+Lawrence%2C+)
16. Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A review of relational machine learning for knowledge graphs: From multi-relational link prediction to automated knowledge graph construction. arXiv preprint arXiv:1503.00759. [find](https://scholar.google.com/scholar?q=Maximilian+Nickel%2C+Kevin+Murphy%2C+Volker+Tresp%2C+and+Evgeniy+Gabrilovich%2E+2015%2E+A+review+of+relational+machine+learning+for+knowledge+graphs%3A+From+multi%2Drelational+link+prediction+to+automated+knowledge)
17. Sebastian Riedel, Limin Yao, Benjamin M. Marlin, and Andrew McCallum. 2013. Relation extraction with matrix factorization and universal schemas. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). [find](https://scholar.google.com/scholar?q=Sebastian+Riedel%2C+Limin+Yao%2C+Benjamin+M%2E+Marlin%2C+and+Andrew+McCallum%2E+2013%2E+Relation+extraction+with+matrix+factorization+and+universal+schemas%2E+In+North+American+Chapter+of+the+Association+for+Comput)
18. Martin Riedmiller and Heinrich Braun. 1993. A direct adaptive method for faster backpropagation learning: The rprop algorithm. In Neural Networks, 1993., IEEE International Conference on, pages 586–591. IEEE. [find](https://scholar.google.com/scholar?q=Martin+Riedmiller+and+Heinrich+Braun%2E+1993%2E+A+direct+adaptive+method+for+faster+backpropagation+learning%3A+The+rprop+algorithm%2E+In+Neural+Networks%2C+1993%2E%2C+IEEE+International+Conference+on%2C+pages+586%E2%80%9359)
19. Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pages 697–706. ACM. [find](https://scholar.google.com/scholar?q=Fabian+M+Suchanek%2C+Gjergji+Kasneci%2C+and+Gerhard+Weikum%2E+2007%2E+Yago%3A+a+core+of+semantic+knowledge%2E+In+Proceedings+of+the+16th+international+conference+on+World+Wide+Web%2C+pages+697%E2%80%93706%2E+ACM%2E)
20. Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pages 1112–1119. [find](https://scholar.google.com/scholar?q=Zhen+Wang%2C+Jianwen+Zhang%2C+Jianlin+Feng%2C+and+Zheng+Chen%2E+2014%2E+Knowledge+graph+embedding+by+translating+on+hyperplanes%2E+In+Proceedings+of+the+Twenty%2DEighth+AAAI+Conference+on+Artificial+Intelligence%2C+p)
21. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations (ICLR). [our card](https://md.research.elamur.ai/free/en/papers/yang-2014-embedding-entities-relations-learning-inference/)

*Updated 23 October 2026*