---
title: "Inducing Relational Knowledge from BERT — Bouraoui 2019"
description: "The paper investigates whether pre-trained language models like BERT store relational knowledge… 34 references, cited by 97 papers here."
authors: ["Zied Bouraoui", "Jose Camacho-Collados", "Steven Schockaert"]
year: 2019
venue: "arXiv:1911.12753"
source: "https://arxiv.org/abs/1911.12753"
doi: "10.48550/arxiv.1911.12753"
arxiv: "1911.12753"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bouraoui-2019-inducing-relational-knowledge-bert/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bouraoui-2019-inducing-relational-knowledge-bert/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Inducing Relational Knowledge from BERT

# Inducing Relational Knowledge from BERT

**Authors:** [Zied Bouraoui](https://md.research.elamur.ai/free/en/authors/zied-bouraoui/), [Jose Camacho-Collados](https://md.research.elamur.ai/free/en/authors/jose-camacho-collados/) and [Steven Schockaert](https://md.research.elamur.ai/free/en/authors/steven-schockaert/) · **Year:** 2019 · **Venue:** [arXiv:1911.12753](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/) ([Proceedings of the AAAI Conference on Artificial Intelligence 2019](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/2019/)) · **Publisher:** [Association for the Advancement of Artificial Intelligence](https://md.research.elamur.ai/free/en/publishers/association-for-the-advancement-of-artificial-intelligence/) · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1911.12753) · [PDF](https://arxiv.org/pdf/1911.12753) · [DOI 10.48550/arxiv.1911.12753](https://doi.org/10.48550/arxiv.1911.12753) · [arXiv 1911.12753](https://arxiv.org/abs/1911.12753) · [OpenAlex](https://openalex.org/W2989867050)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [97](https://md.research.elamur.ai/free/en/papers/bouraoui-2019-inducing-relational-knowledge-bert/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper investigates whether pre-trained language models like BERT store relational knowledge beyond what static word embeddings (Skip-gram, GloVe) already encode. The authors frame this as relation induction: given a few example pairs, decide whether a new pair belongs to the same relation. BERT's answers depend heavily on how a query sentence is worded, so the authors automate the search for good prompts. They collect corpus sentences that mention the seed pairs and treat them as templates. They keep only the templates from which a pre-trained model can recover the masked head or tail word of the seed examples. BERT is then fine-tuned to judge candidate pairs inserted into the retained templates. No corpus sentences about the candidate pair are consulted, so predictions depend solely on what the model has internalized. The method was tested on relations from the Google analogy dataset, BATS and DiffVec. It clearly beats embedding-based methods on encyclopedic and commonsense relations, roughly matches them on lexical ones, and performs poorly on morphological ones. This shows that factual knowledge can be pulled out of BERT without hand-written patterns.

## Key points
- The pipeline has three steps: collect sentences containing seed pairs, filter templates by checking whether BERT can predict the masked head or tail words, and fine-tune BERT as a pair classifier over the instantiated templates.
- Probing BERT-Large-Uncased with short hand-made sentences gives mixed results. It handles capital-of well but fails on colours, and adding a single word such as "often" can flip an answer from sensible to meaningless.
- Automatic templates far outperform short manual ones (F1, hand vs. auto): capital-world 51.4 vs. 89.5 and currency 46.2 vs. 93.6 (Google), item:location 28.6 vs. 54.2 and concealment 25.7 vs. 52.7 (DiffVec).
- Selected templates tend to be long and context-rich. Some give only indirect evidence, for example sentences about embassies located in capitals.
- The method gives strong gains on encyclopedic and commonsense relations, performs on par with word vectors for hypernymy and meronymy, and is unsuitable for morphological relations in BATS.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cardiff University](https://md.research.elamur.ai/free/en/universities/cardiff-university/) — University, Cardiff, United Kingdom · 23 works · 2,549 citations
- [Centre de Recherche en Informatique de Lens](https://md.research.elamur.ai/free/en/labs/centre-de-recherche-en-informatique-de-lens/) — Research institute, Lens, France · 3 works · 105 citations
- [Université d'Artois](https://md.research.elamur.ai/free/en/universities/universite-d-artois/) — University, Arras, France · 3 works · 105 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Zied Bouraoui, Jose Camacho-Collados and Steven Schockaert. 2019. Inducing Relational Knowledge from BERT. arXiv:1911.12753.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bouraoui2019inducing,
  title = {{Inducing Relational Knowledge from BERT}},
  author = {Zied Bouraoui, Jose Camacho-Collados and Steven Schockaert},
  year = {2019},
  doi = {10.48550/arxiv.1911.12753},
  eprint = {1911.12753},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.12753}
}
```

- [← previous HEAD-QA: A Healthcare Dataset for Complex Reasoning](https://md.research.elamur.ai/free/en/papers/vilares-2019-head-qa-healthcare-dataset-complex/)
- [next → Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering](https://md.research.elamur.ai/free/en/papers/ma-2019-generalizable-neuro-symbolic-systems-commonsense/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

34 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allen, C., and Hospedales, T. M. 2019. Analogies explained: Towards understanding word embeddings. In Proc. ICML. [find](https://scholar.google.com/scholar?q=Allen%2C+C%2E%2C+and+Hospedales%2C+T%2E+M%2E+2019%2E+Analogies+explained%3A+Towards+understanding+word+embeddings%2E+In+Proc%2E+ICML%2E)
2. Amrami, A., and Goldberg, Y. 2018. Word sense induction with neural biLM and symmetric patterns. In Proc. EMNLP, 4860–4867. [find](https://scholar.google.com/scholar?q=Amrami%2C+A%2E%2C+and+Goldberg%2C+Y%2E+2018%2E+Word+sense+induction+with+neural+biLM+and+symmetric+patterns%2E+In+Proc%2E+EMNLP%2C+4860%E2%80%934867%2E)
3. Arora, S.; Li, Y.; Liang, Y.; Ma, T.; and Risteski, A. 2016. [A latent variable model approach to pmi-based word embeddings](https://md.research.elamur.ai/free/en/papers/arora-2016-latent-variable-model-approach-pmi/). TACL 4:385–399.
4. Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Celikyilmaz, A.; and Choi, Y. 2019. [COMET: Commonsense transformers for automatic knowledge graph construction](https://md.research.elamur.ai/free/en/papers/bosselut-2019-comet-commonsense-transformers-automatic-knowledge/). In Proc. ACL, 4762–4779.
5. Bouraoui, Z.; Jameel, S.; and Schockaert, S. 2018. Relation induction in word embeddings revisited. In Proc. [find](https://scholar.google.com/scholar?q=Bouraoui%2C+Z%2E%3B+Jameel%2C+S%2E%3B+and+Schockaert%2C+S%2E+2018%2E+Relation+induction+in+word+embeddings+revisited%2E+In+Proc%2E)
6. COLING. Camacho-Collados, J.; Espinosa-Anke, L.; Jameel, S.; and Schockaert, S. 2019. A latent variable model for learning distributional relation vectors. In Proc. [find](https://scholar.google.com/scholar?q=COLING%2E+Camacho%2DCollados%2C+J%2E%3B+Espinosa%2DAnke%2C+L%2E%3B+Jameel%2C+S%2E%3B+and+Schockaert%2C+S%2E+2019%2E+A+latent+variable+model+for+learning+distributional+relation+vectors%2E+In+Proc%2E)
7. IJCAI. Camacho-Collados, J.; Espinosa-Anke, L.; and Schockaert, S. 2019. Relational word embeddings. In Proc. ACL. [find](https://scholar.google.com/scholar?q=IJCAI%2E+Camacho%2DCollados%2C+J%2E%3B+Espinosa%2DAnke%2C+L%2E%3B+and+Schockaert%2C+S%2E+2019%2E+Relational+word+embeddings%2E+In+Proc%2E+ACL%2E)
8. Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proc. NAACL-HLT.
9. Drozd, A.; Gladkova, A.; and Matsuoka, S. 2016. Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen. In Proc. COLING, 3519–3530. [find](https://scholar.google.com/scholar?q=Drozd%2C+A%2E%3B+Gladkova%2C+A%2E%3B+and+Matsuoka%2C+S%2E+2016%2E+Word+embeddings%2C+analogies%2C+and+machine+learning%3A+Beyond+king+%2D+man+%2B+woman+%3D+queen%2E+In+Proc%2E+COLING%2C+3519%E2%80%933530%2E)
10. Espinosa Anke, L., and Schockaert, S. 2018. SeVeN: Augmenting word embeddings with unsupervised relation vectors. In Proc. COLING, 2653–2665. [find](https://scholar.google.com/scholar?q=Espinosa+Anke%2C+L%2E%2C+and+Schockaert%2C+S%2E+2018%2E+SeVeN%3A+Augmenting+word+embeddings+with+unsupervised+relation+vectors%2E+In+Proc%2E+COLING%2C+2653%E2%80%932665%2E)
11. Forbes, M.; Holtzman, A.; and Choi, Y. 2019. [Do neural language representations learn physical commonsense?](https://md.research.elamur.ai/free/en/papers/forbes-2019-do-neural-language-representations-learn/) Proc. of the Conference of the Cognitive Science Society.
12. Gladkova, A.; Drozd, A.; and Matsuoka, S. 2016. [Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t](https://md.research.elamur.ai/free/en/papers/gladkova-2016-analogy-based-detection-morphological-semantic/). In Proc. of the Student Research Workshop at NAACL, 8–15.
13. Goldberg, Y. 2019. [Assessing bert’s syntactic abilities](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). CoRR abs/1901.05287.
14. Gordon, J., and Durme, B. V. 2013. Reporting bias and knowledge acquisition. In Proc. AKBC, 25–30. Gupta, A.; Boleda, G.; Baroni, M.; and Padó, S. 2015. Distributional vectors encode referential attributes. In Proc. EMNLP, 12–21. [find](https://scholar.google.com/scholar?q=Gordon%2C+J%2E%2C+and+Durme%2C+B%2E+V%2E+2013%2E+Reporting+bias+and+knowledge+acquisition%2E+In+Proc%2E+AKBC%2C+25%E2%80%9330%2E+Gupta%2C+A%2E%3B+Boleda%2C+G%2E%3B+Baroni%2C+M%2E%3B+and+Pado%CC%81%2C+S%2E+2015%2E+Distributional+vectors+encode+referential+attr)
15. Hewitt, J., and Manning, C. D. 2019. [A structural probe for finding syntax in word representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/). In Proc. NAACL, 4129–4138.
16. Jameel, S.; Bouraoui, Z.; and Schockaert, S. 2018. Unsupervised learning of distributional relation vectors. In Proc. ACL, 23–33. [find](https://scholar.google.com/scholar?q=Jameel%2C+S%2E%3B+Bouraoui%2C+Z%2E%3B+and+Schockaert%2C+S%2E+2018%2E+Unsupervised+learning+of+distributional+relation+vectors%2E+In+Proc%2E+ACL%2C+23%E2%80%9333%2E)
17. Jawahar, G.; Sagot, B.; and Seddah, D. 2019. [What does BERT learn about the structure of language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) In Proc. ACL, 3651–3657.
18. Joshi, M.; Choi, E.; Levy, O.; Weld, D. S.; and Zettlemoyer, L. 2019. pair2vec: Compositional word-pair embeddings for cross-sentence inference. In Proc. NAACL. [find](https://scholar.google.com/scholar?q=Joshi%2C+M%2E%3B+Choi%2C+E%2E%3B+Levy%2C+O%2E%3B+Weld%2C+D%2E+S%2E%3B+and+Zettlemoyer%2C+L%2E+2019%2E+pair2vec%3A+Compositional+word%2Dpair+embeddings+for+cross%2Dsentence+inference%2E+In+Proc%2E+NAACL%2E)
19. Levy, O.; Goldberg, Y.; and Ramat-Gan, I. 2014. [Linguistic regularities in sparse and explicit word representations](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/). In Proc. CoNLL, 171–180.
20. Linzen, T. 2016. Issues in evaluating semantic spaces using word analogies. In Proc. of the Workshop on Evaluating Vector-Space Representations for NLP, 13–18. [find](https://scholar.google.com/scholar?q=Linzen%2C+T%2E+2016%2E+Issues+in+evaluating+semantic+spaces+using+word+analogies%2E+In+Proc%2E+of+the+Workshop+on+Evaluating+Vector%2DSpace+Representations+for+NLP%2C+13%E2%80%9318%2E)
21. Logan IV, R. L.; Liu, N. F.; Peters, M. E.; Gardner, M.; and Singh, S. 2019. Barack’s wife hillary: Using knowledge graphs for fact-aware language modeling. In Proc. ACL, 5962–5971. [find](https://scholar.google.com/scholar?q=Logan+IV%2C+R%2E+L%2E%3B+Liu%2C+N%2E+F%2E%3B+Peters%2C+M%2E+E%2E%3B+Gardner%2C+M%2E%3B+and+Singh%2C+S%2E+2019%2E+Barack%E2%80%99s+wife+hillary%3A+Using+knowledge+graphs+for+fact%2Daware+language+modeling%2E+In+Proc%2E+ACL%2C+5962%E2%80%935971%2E)
22. Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). In Proc. ICLR.
23. Mikolov, T.; Yih, W.-t.; and Zweig, G. 2013. [Linguistic regularities in continuous space word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2013-linguistic-regularities-continuous-space-word/). In Proc. NAACL, 746–751.
24. Nissim, M.; van Noord, R.; and van der Goot, R. 2019. [Fair is better than sensational:man is to doctor as woman is to doctor](https://md.research.elamur.ai/free/en/papers/nissim-2020-fair-is-better-than-sensational/). arXiv preprint arXiv:1905.09866.
25. Pennington, J.; Socher, R.; and Manning, C. D. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proc. EMNLP, 1532–1543. Petroni, F.; Rocktäschel, T.; Riedel, S.; Lewis, P.; Bakhtin, A.; Wu, Y.; and Miller, A. 2019. Language models as knowledge bases? In Proc. EMNLP, 2463–2473.
26. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language models are unsupervised multitask learners. Technical Report. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%3B+Wu%2C+J%2E%3B+Child%2C+R%2E%3B+Luan%2C+D%2E%3B+Amodei%2C+D%2E%3B+and+Sutskever%2C+I%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+Report%2E)
27. Rogers, A.; Drozd, A.; and Li, B. 2017. The (too many) problems of analogical reasoning with word vectors. In Proc. *SEM, 135–148. [find](https://scholar.google.com/scholar?q=Rogers%2C+A%2E%3B+Drozd%2C+A%2E%3B+and+Li%2C+B%2E+2017%2E+The+%28too+many%29+problems+of+analogical+reasoning+with+word+vectors%2E+In+Proc%2E+%2ASEM%2C+135%E2%80%93148%2E)
28. Rubinstein, D.; Levi, E.; Schwartz, R.; and Rappoport, A. 2015. How well do distributional models capture different types of semantic knowledge? In Proc. ACL, 726–730. [find](https://scholar.google.com/scholar?q=Rubinstein%2C+D%2E%3B+Levi%2C+E%2E%3B+Schwartz%2C+R%2E%3B+and+Rappoport%2C+A%2E+2015%2E+How+well+do+distributional+models+capture+different+types+of+semantic+knowledge%3F+In+Proc%2E+ACL%2C+726%E2%80%93730%2E)
29. Tenney, I.; Das, D.; and Pavlick, E. 2019. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proc. COLING, 4593–4601.
30. Vu, T., and Shwartz, V. 2018. Integrating multiplicative features into supervised distributional methods for lexical entailment. In Proc. *SEM, 160–166. Vulić, I., and Mrkšić, N. 2018. [Specialising word vectors for lexical entailment](https://md.research.elamur.ai/free/en/papers/vulic-2018-specialising-word-vectors-lexical-entailment/). In Proc. NAACL, 1134–1145.
31. Vylomova, E.; Rimell, L.; Cohn, T.; and Baldwin, T. 2016. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Vylomova%2C+E%2E%3B+Rimell%2C+L%2E%3B+Cohn%2C+T%2E%3B+and+Baldwin%2C+T%2E+2016%2E+Take+and+took%2C+gaggle+and+goose%2C+book+and+read%3A+Evaluating+the+utility+of+vector+differences+for+lexical+relation+learning%2E+In+Proc%2E+ACL%2E)
32. Washio, K., and Kato, T. 2018a. Filling missing paths: Modeling co-occurrences of word pairs and dependency paths for recognizing lexical semantic relations. In Proc. NAACL. [find](https://scholar.google.com/scholar?q=Washio%2C+K%2E%2C+and+Kato%2C+T%2E+2018a%2E+Filling+missing+paths%3A+Modeling+co%2Doccurrences+of+word+pairs+and+dependency+paths+for+recognizing+lexical+semantic+relations%2E+In+Proc%2E+NAACL%2E)
33. Washio, K., and Kato, T. 2018b. Neural latent relational analysis to capture lexical semantic relations in a vector space. In Proc. EMNLP, 594–600. [find](https://scholar.google.com/scholar?q=Washio%2C+K%2E%2C+and+Kato%2C+T%2E+2018b%2E+Neural+latent+relational+analysis+to+capture+lexical+semantic+relations+in+a+vector+space%2E+In+Proc%2E+EMNLP%2C+594%E2%80%93600%2E)
34. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.; and Le, Q. V. 2019. [XLNet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). arXiv:1906.08237.

*Updated 2 November 2026*