---
title: "Matching the Blanks: Distributional Similarity for Relation Learning — Soares 2019"
description: "Next, they propose a pre-training method, \"matching the blanks\" (MTB). 80.4% accuracy. 31 references, cited by 294 papers here."
authors: ["Livio Baldini Soares", "Nicholas FitzGerald", "Jeffrey Ling", "Tom Kwiatkowski"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1279/"
doi: "10.18653/v1/p19-1279"
arxiv: "1906.03158"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/soares-2019-matching-blanks-distributional-similarity-relation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/soares-2019-matching-blanks-distributional-similarity-relation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Matching the Blanks: Distributional Similarity for Relation Learning

# Matching the Blanks: Distributional Similarity for Relation Learning

**Authors:** [Livio Baldini Soares](https://md.research.elamur.ai/free/en/authors/livio-baldini-soares/), [Nicholas FitzGerald](https://md.research.elamur.ai/free/en/authors/nicholas-fitzgerald/), Jeffrey Ling and [Tom Kwiatkowski](https://md.research.elamur.ai/free/en/authors/tom-kwiatkowski/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1279/) · [PDF](https://aclanthology.org/P19-1279.pdf) · [DOI 10.18653/v1/p19-1279](https://doi.org/10.18653/v1/p19-1279) · [arXiv 1906.03158](https://arxiv.org/abs/1906.03158)

**Citations:** [294](https://md.research.elamur.ai/free/en/papers/soares-2019-matching-blanks-distributional-similarity-relation/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary

The paper addresses the task of extracting relations between entities in text. The authors aim to obtain general-purpose relation representations that do not depend on a fixed schema or a large knowledge graph. First, using BERT LARGE, they compare ways to mark the two target entities in the input and to obtain a fixed-length relation vector from the output. The best approach was to surround each entity with special marker tokens and to use the concatenation of the states at the start markers as the relation representation. Next, they propose a pre-training method, "matching the blanks" (MTB). It uses only text with linked entities (entity linking), in which entity mentions are sometimes replaced with the symbol [BLANK]. The model learns to produce similar vectors for passages that concern the same pair of entities. This requires neither manual annotation nor a knowledge graph. The main result: on the FewRel few-shot task, the model surpasses the previous best result even without any FewRel training data. After fine-tuning, it exceeds human accuracy. MTB also improves results on SemEval 2010 Task 8, KBP37 and TACRED. This substantially reduces the amount of human annotation needed to build relation extraction systems.

## Key points

- **Architecture.** The best variant, BERT EM (ENTITY MARKERS + ENTITY START), with supervised training alone achieves test F1 of 89.2 on SemEval 2010 Task 8, 68.3 on KBP37 and 70.1 on TACRED. This is higher than the previously published results: 88.0, 58.8 and 68.2, respectively.
- **FewRel without training data.** BERT EM + MTB achieves 80.4% accuracy in 5-way-1-shot and 71.5% in 10-way-1-shot. The Prototypical Network (CNN) from Han et al. (2018) achieved 71.6% and 58.8%, so the gain is 8.8 and 12.7 points.
- **Annotation savings.** If all 64 relation types are kept, BERT EM + MTB needs only about 6% of the FewRel training data to reach the level of BERT EM fine-tuned on the full set. Diversity of relation types is more useful than many examples per type.
- **Little data.** MTB helps most when fine-tuning data is scarce. For example, with 1% of the TACRED training set, F1 rises from 32.8 to 43.4, and with 1% of KBP37, from 40.1 to 44.2.
- **Authors' plans.** They intend to discover new relations by clustering BERT EM + MTB vectors and to use such representations for distributed knowledge bases.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/)

## Cite

Cite as: Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling and Tom Kwiatkowski. 2019. Matching the Blanks: Distributional Similarity for Relation Learning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{soares2019matching,
  title = {{Matching the Blanks: Distributional Similarity for Relation Learning}},
  author = {Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling and Tom Kwiatkowski},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1279},
  eprint = {1906.03158},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1279}
}
```

- [← previous Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing](https://md.research.elamur.ai/free/en/papers/schuster-2019-cross-lingual-alignment-contextual-word/)
- [next → FlauBERT: Unsupervised Language Model Pre-training for French](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Langu… [our card](https://md.research.elamur.ai/free/en/papers/angeli-2015-leveraging-linguistic-structure-open-domain/)
2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
3. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI’07, pages 2670– 2676, San Francisco, CA, USA. Morgan Kaufmann Publisher… [our card](https://md.research.elamur.ai/free/en/papers/banko-2007-open-information-extraction-web/)
4. Ivan Bilan and Benjamin Roth. 2018. Position-aware self-attention with relative positional encodings for slot filling. CoRR, abs/1807.03052. — [arXiv:1807.03052](https://arxiv.org/abs/1807.03052) [find](https://scholar.google.com/scholar?q=Ivan+Bilan+and+Benjamin+Roth%2E+2018%2E+Position%2Daware+self%2Dattention+with+relative+positional+encodings+for+slot+filling%2E+CoRR%2C+abs%2F1807%2E03052%2E+%E2%80%94+%5BarXiv%3A1807%2E03052%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1807%2E03052%29)
5. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, pages 1247–1250, New York, NY, USA.… [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+A+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+the+2008+ACM+SI)
6. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, p… [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Nicolas+Usunier%2C+Alberto+Garcia%2DDuran%2C+Jason+Weston%2C+and+Oksana+Yakhnenko%2E+2013%2E+Translating+embeddings+for+modeling+multi%2Drelational+data%2E+In+C%2E+J%2E+C%2E+Burges%2C+L%2E+Bottou%2C+M%2E+Welling%2C+Z)
7. Claire Cardie. 1997. Empirical methods in information extraction. AI Magazine, 18(4):65–80. [find](https://scholar.google.com/scholar?q=Claire+Cardie%2E+1997%2E+Empirical+methods+in+information+extraction%2E+AI+Magazine%2C+18%284%29%3A65%E2%80%9380%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
9. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545, Edinburgh, Scotland, UK. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anthony+Fader%2C+Stephen+Soderland%2C+and+Oren+Etzioni%2E+2011%2E+Identifying+relations+for+open+information+extraction%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
10. Zhou GuoDong, Su Jian, Zhang Jie, and Zhang Min. 2005. Exploring various knowledge in relation extraction. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 427–434. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/zhou-2005-exploring-various-knowledge-relation-extraction/)
11. Michael U Gutmann and Aapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13(Feb):307–361. [find](https://scholar.google.com/scholar?q=Michael+U+Gutmann+and+Aapo+Hyva%CC%88rinen%2E+2012%2E+Noise%2Dcontrastive+estimation+of+unnormalized+statistical+models%2C+with+applications+to+natural+image+statistics%2E+Journal+of+Machine+Learning+Research%2C+13%28Fe)
12. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803– 48… [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Hao+Zhu%2C+Pengfei+Yu%2C+Ziyun+Wang%2C+Yuan+Yao%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2018%2E+Fewrel%3A+A+large%2Dscale+supervised+few%2Dshot+relation+classification+dataset+with+state%2Dof%2Dthe%2Dart+evaluation%2E+In+Pr)
13. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. [find](https://scholar.google.com/scholar?q=Zellig+S+Harris%2E+1954%2E+Distributional+structure%2E+Word%2C+10%282%2D3%29%3A146%E2%80%93162%2E)
14. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Seman… [our card](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/)
15. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1… [find](https://scholar.google.com/scholar?q=Raphael+Hoffmann%2C+Congle+Zhang%2C+Xiao+Ling%2C+Luke+Zettlemoyer%2C+and+Daniel+S+Weld%2E+2011%2E+Knowledge%2Dbased+weak+supervision+for+information+extraction+of+overlapping+relations%2E+In+Proceedings+of+the+49th+A)
16. Nanda Kambhatla. 2004. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. In Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, page 22. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nanda+Kambhatla%2E+2004%2E+Combining+lexical%2C+syntactic%2C+and+semantic+features+with+maximum+entropy+models+for+extracting+relations%2E+In+Proceedings+of+the+ACL+2004+on+Interactive+poster+and+demonstration+)
17. Dekang Lin and Patrick Pantel. 2001. DIRT: Discovery of Inference Rules from Text. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’01), pages 323–328, New York, NY, USA. ACM Press. [find](https://scholar.google.com/scholar?q=Dekang+Lin+and+Patrick+Pantel%2E+2001%2E+DIRT%3A+Discovery+of+Inference+Rules+from+Text%2E+In+Proceedings+of+the+Seventh+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining+%28KDD%E2%80%9901%29%2C+pa)
18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
19. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages… [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Daniel+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+t)
20. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in neural information processing systems, pages 2265–2273. [find](https://scholar.google.com/scholar?q=Andriy+Mnih+and+Koray+Kavukcuoglu%2E+2013%2E+Learning+word+embeddings+efficiently+with+noise%2Dcontrastive+estimation%2E+In+Advances+in+neural+information+processing+systems%2C+pages+2265%E2%80%932273%2E)
21. Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016. Holographic embeddings of knowledge graphs. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pages 1955–1961. AAAI Press. [find](https://scholar.google.com/scholar?q=Maximilian+Nickel%2C+Lorenzo+Rosasco%2C+and+Tomaso+Poggio%2E+2016%2E+Holographic+embeddings+of+knowledge+graphs%2E+In+Proceedings+of+the+Thirtieth+AAAI+Conference+on+Artificial+Intelligence%2C+AAAI%E2%80%9916%2C+pages+1955)
22. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
23. Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74–… [find](https://scholar.google.com/scholar?q=Sebastian+Riedel%2C+Limin+Yao%2C+Andrew+McCallum%2C+and+Benjamin+M+Marlin%2E+2013%2E+Relation+extraction+with+matrix+factorization+and+universal+schemas%2E+In+Proceedings+of+the+2013+Conference+of+the+North+Ameri)
24. Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 885– 895,… [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky%2C+Julian+Michael%2C+Luke+Zettlemoyer%2C+and+Ido+Dagan%2E+2018%2E+Supervised+open+information+extraction%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association)
25. Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509, Lisbon, Portugal. A… [our card](https://md.research.elamur.ai/free/en/papers/toutanova-2015-representing-text-joint-embedding-text/)
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
27. Patrick Verga and Andrew McCallum. 2016. Row-less universal schema. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, pages 63–68, San Diego, CA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Patrick+Verga+and+Andrew+McCallum%2E+2016%2E+Row%2Dless+universal+schema%2E+In+Proceedings+of+the+5th+Workshop+on+Automated+Knowledge+Base+Construction%2C+pages+63%E2%80%9368%2C+San+Diego%2C+CA%2E+Association+for+Computation)
28. Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016. Relation classification via multi-level attention cnns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1298–1307. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/wang-2016-relation-classification-multi-level-attention/)
29. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335–2344. Dublin City University and Associatio… [our card](https://md.research.elamur.ai/free/en/papers/zeng-2014-relation-classification-convolutional-deep-neural/)
30. Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. CoRR, abs/1508.01006. — [arXiv:1508.01006](https://arxiv.org/abs/1508.01006) [find](https://scholar.google.com/scholar?q=Dongxu+Zhang+and+Dong+Wang%2E+2015%2E+Relation+classification+via+recurrent+neural+network%2E+CoRR%2C+abs%2F1508%2E01006%2E+%E2%80%94+%5BarXiv%3A1508%2E01006%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1508%2E01006%29)
31. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35–45. [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Victor+Zhong%2C+Danqi+Chen%2C+Gabor+Angeli%2C+and+Christopher+D+Manning%2E+2017%2E+Position%2Daware+attention+and+supervised+data+improve+slot+filling%2E+In+Proceedings+of+the+2017+Conference+on+Empiri)