---
title: "Zero-Shot Relation Extraction via Reading Comprehension (2017)"
description: "The authors propose treating relation extraction as a reading comprehension task. 16 references, cited by 346 papers here."
authors: ["Omer Levy", "Minjoon Seo", "Eunsol Choi", "Luke Zettlemoyer"]
year: 2017
venue: "Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)"
source: "https://aclanthology.org/K17-1034/"
doi: "10.18653/v1/k17-1034"
arxiv: "1706.04115"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/levy-2017-zero-shot-relation-extraction-reading/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/levy-2017-zero-shot-relation-extraction-reading/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Zero-Shot Relation Extraction via Reading Comprehension

# Zero-Shot Relation Extraction via Reading Comprehension

**Authors:** [Omer Levy](https://md.research.elamur.ai/free/en/authors/omer-levy/), [Minjoon Seo](https://md.research.elamur.ai/free/en/authors/minjoon-seo/), [Eunsol Choi](https://md.research.elamur.ai/free/en/authors/eunsol-choi/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2017 · **Venue:** [Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)](https://md.research.elamur.ai/free/en/venues/conll/) ([CoNLL 2017](https://md.research.elamur.ai/free/en/venues/conll/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/K17-1034/) · [PDF](https://aclanthology.org/K17-1034.pdf) · [DOI 10.18653/v1/k17-1034](https://doi.org/10.18653/v1/k17-1034) · [arXiv 1706.04115](https://arxiv.org/abs/1706.04115) · [OpenAlex](https://openalex.org/W2624677889)

**Citations:** 460 citations (OpenAlex, 2026-10-14) · [346](https://md.research.elamur.ai/free/en/papers/levy-2017-zero-shot-relation-extraction-reading/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

The authors propose treating relation extraction as a reading comprehension task. Each relation type from a knowledge base is described by one or more templates of natural-language questions with a slot for an entity. The model answers such a question with respect to a specific sentence and thereby fills the relation slot. If the sentence contains no answer, the model must recognize this. For training, the authors used data from WikiReading, obtained through distant supervision for Wikidata relations. The question templates were collected and verified via crowdsourcing; annotation was done for relation types rather than for individual examples, so it was inexpensive. The neural reading comprehension model of Seo et al. (2016), known as BiDAF, was extended with the ability to return an empty answer. The main advantage of the approach lies in the zero-shot setting: a new relation that the model did not see during training can be specified at test time with just a few questions. For new question phrasings of already known relations, performance barely drops. For entirely new relations, the result is more modest but no longer zero. This opens the way to flexible knowledge base population without separate annotation for each relation.

## Key points

- **Data:** 120 Wikidata relations from the WikiReading corpus. For each, there are on average about 10 crowdsourced question templates. In total, this amounts to over 30M "question – sentence – answer" examples.
- **Known relations with new question phrasings:** a relative F1 drop of only about 4%.
- **Relations unseen during training (zero-shot):** F1 rises from nearly zero to 41%.
- **Analysis of 100 examples:** cues about the answer type occur more often than cues about the relation itself (58% vs. 42%). The model makes better use of general type cues that recur across many relations than of cues specific to a single relation.
- **Resource:** the authors have made the code and data publicly available.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 275 works · 97,634 citations
- [Bar-Ilan University](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/) — University, Ramat Gan, Israel · 141 works · 23,225 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works · [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 167 works · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) · 192 works

💡 Concepts (1): [💡 Few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) · 45 works

## Cite

Cite as: Omer Levy, Minjoon Seo, Eunsol Choi and Luke Zettlemoyer. 2017. Zero-Shot Relation Extraction via Reading Comprehension. Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{levy2017zero,
  title = {{Zero-Shot Relation Extraction via Reading Comprehension}},
  author = {Omer Levy, Minjoon Seo, Eunsol Choi and Luke Zettlemoyer},
  booktitle = {Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)},
  year = {2017},
  doi = {10.18653/v1/k17-1034},
  eprint = {1706.04115},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/k17-1034}
}
```

- [← previous Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe](https://md.research.elamur.ai/free/en/papers/straka-2017-tokenizing-pos-tagging-lemmatizing-parsing/)
- [next → Semi-supervised sequence tagging with bidirectional language models](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/)

## References

16 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. [Open information extraction from the web](https://md.research.elamur.ai/free/en/papers/banko-2007-open-information-extraction-web/). In Proceedings of the 20th International Joint Conference on Artifical Intelligence. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, IJCAI’07, pages 2670…
2. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1… [find](https://scholar.google.com/scholar?q=Raphael+Hoffmann%2C+Congle+Zhang%2C+Xiao+Ling%2C+Luke+Zettlemoyer%2C+and+Daniel+S+Weld%2E+2011%2E+Knowledge%2Dbased+weak+supervision+for+information+extraction+of+overlapping+relations%2E+In+Proceedings+of+the+49th+A)
3. Kenton Lee, Tom Kwiatkowski, Ankur Parikh, and Dipanjan Das. 2016. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436 . — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Tom+Kwiatkowski%2C+Ankur+Parikh%2C+and+Dipanjan+Das%2E+2016%2E+Learning+recurrent+span+representations+for+extractive+question+answering%2E+arXiv+preprint+arXiv%3A1611%2E01436+%2E+%E2%80%94+%5BarXiv%3A1611%2E01436%5D%28htt)
4. Angli Liu, Stephen Soderland, Jonathan Bragg, Christopher H. Lin, Xiao Ling, and Daniel S. Weld. 2016. Effective crowd annotation for relation extraction. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… [find](https://scholar.google.com/scholar?q=Angli+Liu%2C+Stephen+Soderland%2C+Jonathan+Bragg%2C+Christopher+H%2E+Lin%2C+Xiao+Ling%2C+and+Daniel+S%2E+Weld%2E+2016%2E+Effective+crowd+annotation+for+relation+extraction%2E+In+Proceedings+of+the+2016+Conference+of+the+)
5. Makoto Miwa and Mohit Bansal. 2016. [End-to-end relation extraction using lstms on sequences and tree structures](https://md.research.elamur.ai/free/en/papers/miwa-2016-end-end-relation-extraction-lstms/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1105–…
6. Patrick Verga, Arvind Neelakantan, and Andrew Mc-Callum. 2017. Generalizing to unseen entities and entity pairs with row-less universal schema. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Compu… [find](https://scholar.google.com/scholar?q=Patrick+Verga%2C+Arvind+Neelakantan%2C+and+Andrew+Mc%2DCallum%2E+2017%2E+Generalizing+to+unseen+entities+and+entity+pairs+with+row%2Dless+universal+schema%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Ch)
7. Alvaro Morales, Varot Premtoon, Cordelia Avery, Sue Felshin, and Boris Katz. 2016. Learning to answer questions from wikipedia infoboxes. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1930–19… [find](https://scholar.google.com/scholar?q=Alvaro+Morales%2C+Varot+Premtoon%2C+Cordelia+Avery%2C+Sue+Felshin%2C+and+Boris+Katz%2E+2016%2E+Learning+to+answer+questions+from+wikipedia+infoboxes%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+)
8. Denny Vrandečić. 2012. Wikidata: A new platform for collaborative data collection. In Proceedings of the 21st international conference companion on World Wide Web. ACM, pages 1063–1064. [find](https://scholar.google.com/scholar?q=Denny+Vrandec%CC%8Cic%CC%81%2E+2012%2E+Wikidata%3A+A+new+platform+for+collaborative+data+collection%2E+In+Proceedings+of+the+21st+international+conference+companion+on+World+Wide+Web%2E+ACM%2C+pages+1063%E2%80%931064%2E)
9. Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211 . — [arXiv:1612.04211](https://arxiv.org/abs/1612.04211) [find](https://scholar.google.com/scholar?q=Zhiguo+Wang%2C+Haitao+Mi%2C+Wael+Hamza%2C+and+Radu+Florian%2E+2016%2E+Multi%2Dperspective+context+matching+for+machine+comprehension%2E+arXiv+preprint+arXiv%3A1612%2E04211+%2E+%E2%80%94+%5BarXiv%3A1612%2E04211%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
10. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 1532–1543. [http://www.acl…](http://www.acl…) [↗](http://www.acl…)
11. Caiming Xiong, Victor Zhong, and Richard Socher. 2016. [Dynamic coattention networks for question answering](https://md.research.elamur.ai/free/en/papers/xiong-2016-dynamic-coattention-networks-question-answering/). arXiv preprint arXiv:1611.01604 .
12. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the Conference of the Empirical Methods in Natural Language Processing.
13. Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Associat… [find](https://scholar.google.com/scholar?q=Sebastian+Riedel%2C+Limin+Yao%2C+Andrew+McCallum%2C+and+Benjamin+M%2E+Marlin%2E+2013%2E+Relation+extraction+with+matrix+factorization+and+universal+schemas%2E+In+Proceedings+of+the+2013+Conference+of+the+North+Amer)
14. Tim Rocktäschel, Sameer Singh, and Sebastian Riedel. 2015. Injecting logical background knowledge into embeddings for relation extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association f… [find](https://scholar.google.com/scholar?q=Tim+Rockta%CC%88schel%2C+Sameer+Singh%2C+and+Sebastian+Riedel%2E+2015%2E+Injecting+logical+background+knowledge+into+embeddings+for+relation+extraction%2E+In+Proceedings+of+the+2015+Conference+of+the+North+American+)
15. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). arXiv preprint arXiv:1611.01603 .
16. Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Danqi+Chen%2C+Patrick+Pantel%2C+Hoifung+Poon%2C+Pallavi+Choudhury%2C+and+Michael+Gamon%2E+2015%2E+Representing+text+for+joint+embedding)

*Updated 29 October 2026*