---
title: "Distant supervision for relation extraction without labeled data"
description: "The authors propose a distant supervision scheme in which the Freebase knowledge base serves… 27 references, cited by 729 papers here."
authors: ["Mike Mintz", "Steven Bills", "Rion Snow", "Daniel Jurafsky"]
year: 2009
venue: "Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP"
source: "https://aclanthology.org/P09-1113/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mintz-2009-distant-supervision-relation-extraction-without/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mintz-2009-distant-supervision-relation-extraction-without/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/) / Distant supervision for relation extraction without labeled data

# Distant supervision for relation extraction without labeled data

**Authors:** Mike Mintz, [Steven Bills](https://md.research.elamur.ai/free/en/authors/steven-bills/), [Rion Snow](https://md.research.elamur.ai/free/en/authors/rion-snow/) and [Daniel Jurafsky](https://md.research.elamur.ai/free/en/authors/dan-jurafsky/) · **Year:** 2009 · **Venue:** [Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2009](https://md.research.elamur.ai/free/en/venues/acl/2009/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P09-1113/) · [PDF](https://aclanthology.org/P09-1113.pdf) · [OpenAlex](https://openalex.org/W2107598941)

**Citations:** 2,752 citations (OpenAlex, 2026-10-14) · [729](https://md.research.elamur.ai/free/en/papers/mintz-2009-distant-supervision-relation-extraction-without/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses the automatic extraction of semantic relations between entities from text. Classical methods require a manually annotated corpus (e.g., ACE), so data are scarce and performance degrades outside the subject domain. The authors propose a distant supervision scheme in which the Freebase knowledge base serves as the teacher. If two entities are linked by some relation in Freebase, then every sentence in which both occur is treated as a likely example of that relation. Entities in 1.2 million Wikipedia articles are identified with an NER tagger. Features from all sentences in which the same pair occurs are combined into a single vector, and a multiclass logistic regression is then trained on lexical and syntactic (dependency) features. The system extracted 10,000 new facts for 102 relations with a precision of 67.6%. The approach requires no manual annotation, yields relations with the canonical names of the knowledge base, and scales to corpora of any size. It subsequently became the baseline for an entire line of research in relation extraction.

## Key points
- Data: the July 2008 Freebase dump contained 116 million facts, 7,300 relations, and 9 million entities. After filtering, 1.8 million instances of 102 relations among 940 thousand entities remained. For comparison, the ACE corpora contain about 17 thousand manually annotated instances.
- The model is a multiclass logistic regression. It combines about 400,000 noisy features collected from all sentences in which an entity pair occurs.
- Performance was evaluated in two ways: automatically, on a held-out portion of Freebase, and manually, by Amazon Mechanical Turk annotators. Among the top 100 results for each of the 10 most frequent relations, the average precision is 0.67 for syntactic features, 0.66 for lexical features, and 0.69 for both combined.
- Syntactic features help most with ambiguous relations (director–film, writer–film). They are also useful when the entities in a sentence are separated by many words but are adjacent in the dependency tree.
- Limitations: evaluation on held-out Freebase data underestimates performance because of false negatives. As future work, the authors mention chunk features instead of full parsing, as well as coreference resolution.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 167 works · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) · 192 works

💡 Concepts (2): [💡 Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) · 131 works · [💡 Information extraction](https://md.research.elamur.ai/free/en/concepts/information-extraction/) · 50 works

## Cite

Cite as: Mike Mintz, Steven Bills, Rion Snow and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mintz2009distant,
  title = {{Distant supervision for relation extraction without labeled data}},
  author = {Mike Mintz, Steven Bills, Rion Snow and Daniel Jurafsky},
  booktitle = {Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP},
  year = {2009},
  doi = {10.3115/1690219.1690287},
  url = {https://doi.org/10.3115/1690219.1690287}
}
```

- [← previous Active Learning Literature Survey](https://md.research.elamur.ai/free/en/papers/settles-2009-active-learning-literature-survey/)
- [next → Reading Tea Leaves: How Humans Interpret Topic Models](https://md.research.elamur.ai/free/en/papers/chang-2009-reading-tea-leaves-humans-interpret-topic/)
- [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/)

## References

27 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the 5th ACM International Conference on Digital Libraries. [find](https://scholar.google.com/scholar?q=Eugene+Agichtein+and+Luis+Gravano%2E+2000%2E+Snowball%3A+Extracting+relations+from+large+plain%2Dtext+collections%2E+In+Proceedings+of+the+5th+ACM+International+Conference+on+Digital+Libraries%2E)
2. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. [Open information extraction from the web](https://md.research.elamur.ai/free/en/papers/banko-2007-open-information-extraction-web/). In Manuela M Veloso, editor, IJCAI-07, pages 2670– 2676.
3. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD ’08, pages 1247– 1250, New York, NY. ACM. [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+a+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+SIGMOD+%E2%80%9908%2C+pages+1247%E2%80%93+1250%2C+)
4. Sergei Brin. 1998. Extracting patterns and relations from the World Wide Web. In Proceedings World Wide Web and Databases International Workshop, Number 1590 in LNCS, pages 172–183. Springer. [find](https://scholar.google.com/scholar?q=Sergei+Brin%2E+1998%2E+Extracting+patterns+and+relations+from+the+World+Wide+Web%2E+In+Proceedings+World+Wide+Web+and+Databases+International+Workshop%2C+Number+1590+in+LNCS%2C+pages+172%E2%80%93183%2E+Springer%2E)
5. Razvan Bunescu and Raymond Mooney. 2007. [Learning to extract relations from the web using minimal supervision](https://md.research.elamur.ai/free/en/papers/bunescu-2007-learning-extract-relations-web-minimal/). In ACL-07, pages 576–583, Prague, Czech Republic, June.
6. Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In Thomas Lengauer, Reinhard Schneider, Peer Bork, Douglas L. Brutlag, Janice I. Glasgow, Hans W. Mewes, and Ralf Zimmer, editors, ISMB, pages 77–86. AAAI. [find](https://scholar.google.com/scholar?q=Mark+Craven+and+Johan+Kumlien%2E+1999%2E+Constructing+biological+knowledge+bases+by+extracting+information+from+text+sources%2E+In+Thomas+Lengauer%2C+Reinhard+Schneider%2C+Peer+Bork%2C+Douglas+L%2E+Brutlag%2C+Janice+)
7. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. [The Automatic Content Extraction (ACE) Program–Tasks, Data, and Evaluation](https://md.research.elamur.ai/free/en/papers/doddington-2004-automatic-content-extraction-ace-program/). LREC-04, pages 837–840.
8. Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S. Weld, and Alexander Yates. 2005. Unsupervised named-entity extraction from the web: An experimental study. Artificial Intelligence, 165(1):91–134. [find](https://scholar.google.com/scholar?q=Oren+Etzioni%2C+Michael+Cafarella%2C+Doug+Downey%2C+Ana%2DMaria+Popescu%2C+Tal+Shaked%2C+Stephen+Soderland%2C+Daniel+S%2E+Weld%2C+and+Alexander+Yates%2E+2005%2E+Unsupervised+named%2Dentity+extraction+from+the+web%3A+An+experim)
9. Jenny R. Finkel, Trond Grenager, and Christopher Manning. 2005. [Incorporating non-local information into information extraction systems by gibbs sampling](https://md.research.elamur.ai/free/en/papers/finkel-2005-incorporating-non-local-information-into/). In ACL-05, pages 363–370, Ann Arbor, MI.
10. Roxana Girju, Adriana Badulescu, and Dan Moldovan. 2003. [Learning semantic constraints for the automatic discovery of part-whole relations](https://md.research.elamur.ai/free/en/papers/girju-2003-learning-semantic-constraints-automatic-discovery/). In HLT-NAACL-03, pages 1–8, Edmonton, Canada.
11. Marti A. Hearst. 1992. [Automatic acquisition of hyponyms from large text corpora](https://md.research.elamur.ai/free/en/papers/hearst-1992-automatic-acquisition-hyponyms-large-text/). In COLING-92, Nantes, France.
12. Dekang Lin and Patrick Pantel. 2001. Discovery of inference rules for question-answering. Natural Language Engineering, 7(4):343–360. [find](https://scholar.google.com/scholar?q=Dekang+Lin+and+Patrick+Pantel%2E+2001%2E+Discovery+of+inference+rules+for+question%2Danswering%2E+Natural+Language+Engineering%2C+7%284%29%3A343%E2%80%93360%2E)
13. Dekang Lin. 1998. Dependency-based evaluation of minipar. In Workshop on the Evaluation of Parsing Systems. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1998%2E+Dependency%2Dbased+evaluation+of+minipar%2E+In+Workshop+on+the+Evaluation+of+Parsing+Systems%2E)
14. Metaweb. 2008. Freebase data dumps. [http://download.freebase.com/datadumps/](http://download.freebase.com/datadumps/). [↗](http://download.freebase.com/datadumps/)
15. Alexander A. Morgan, Lynette Hirschman, Marc Colosimo, Alexander S. Yeh, and Jeff B. Colombe. 2004. Gene name identification and normalization using a model organism database. J. of Biomedical Informatics, 37(6):396–410. [find](https://scholar.google.com/scholar?q=Alexander+A%2E+Morgan%2C+Lynette+Hirschman%2C+Marc+Colosimo%2C+Alexander+S%2E+Yeh%2C+and+Jeff+B%2E+Colombe%2E+2004%2E+Gene+name+identification+and+normalization+using+a+model+organism+database%2E+J%2E+of+Biomedical+Informa)
16. Patrick Pantel and Marco Pennacchiotti. 2006. [Espresso: leveraging generic patterns for automatically harvesting semantic relations](https://md.research.elamur.ai/free/en/papers/pantel-2006-espresso-leveraging-generic-patterns-automatically/). In COL-ING/ACL 2006, pages 113–120, Sydney, Australia.
17. Marco Pennacchiotti and Patrick Pantel. 2006. A bootstrapping algorithm for automatically harvesting semantic relations. In in Proceedings of Inference in Computational Semantics (ICoS-06), pages 87–96. [find](https://scholar.google.com/scholar?q=Marco+Pennacchiotti+and+Patrick+Pantel%2E+2006%2E+A+bootstrapping+algorithm+for+automatically+harvesting+semantic+relations%2E+In+in+Proceedings+of+Inference+in+Computational+Semantics+%28ICoS%2D06%29%2C+pages+87%E2%80%939)
18. Deepak Ravichandran and Eduard H. Hovy. 2002. [Learning surface text patterns for a question answering system](https://md.research.elamur.ai/free/en/papers/ravichandran-2002-learning-surface-text-patterns-question/). In ACL-02, pages 41–47, Philadelphia, PA.
19. Ellen Riloff and Rosie Jones. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In AAAI-99, pages 474–479. [find](https://scholar.google.com/scholar?q=Ellen+Riloff+and+Rosie+Jones%2E+1999%2E+Learning+dictionaries+for+information+extraction+by+multi%2Dlevel+bootstrapping%2E+In+AAAI%2D99%2C+pages+474%E2%80%93479%2E)
20. Benjamin Rozenfeld and Ronen Feldman. 2008. Self-supervised relation extraction from the web. Knowledge and Information Systems, 17(1):17–33. [find](https://scholar.google.com/scholar?q=Benjamin+Rozenfeld+and+Ronen+Feldman%2E+2008%2E+Self%2Dsupervised+relation+extraction+from+the+web%2E+Knowledge+and+Information+Systems%2C+17%281%29%3A17%E2%80%9333%2E)
21. Yusuke Shinyama and Satoshi Sekine. 2006. [Preemptive information extraction using unrestricted relation discovery](https://md.research.elamur.ai/free/en/papers/shinyama-2006-preemptive-information-extraction-unrestricted-relation/). In HLT-NAACL-06, pages 304–311, New York, NY.
22. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Lawrence K. Saul, Yair Weiss, and Léon Bottou, editors, NIPS 17, pages 1297–1304. MIT Press. [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Daniel+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2005%2E+Learning+syntactic+patterns+for+automatic+hypernym+discovery%2E+In+Lawrence+K%2E+Saul%2C+Yair+Weiss%2C+and+Le%CC%81on+Bottou%2C+editors%2C+NIPS+17%2C+pages+1297%E2%80%931304%2E+)
23. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. [Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/). In EMNLP 2008, pages 254–263, Honolulu, HI.
24. Mihai Surdeanu and Massimiliano Ciaramita. 2007. Robust information extraction with perceptrons. In Proceedings of the NIST 2007 Automatic Content Extraction Workshop (ACE07), March. [find](https://scholar.google.com/scholar?q=Mihai+Surdeanu+and+Massimiliano+Ciaramita%2E+2007%2E+Robust+information+extraction+with+perceptrons%2E+In+Proceedings+of+the+NIST+2007+Automatic+Content+Extraction+Workshop+%28ACE07%29%2C+March%2E)
25. Fei Wu and Daniel S. Weld. 2007. Autonomously se-mantifying wikipedia. In CIKM ’07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pages 41– 50, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Fei+Wu+and+Daniel+S%2E+Weld%2E+2007%2E+Autonomously+se%2Dmantifying+wikipedia%2E+In+CIKM+%E2%80%9907%3A+Proceedings+of+the+sixteenth+ACM+conference+on+Conference+on+information+and+knowledge+management%2C+pages+41%E2%80%93+50%2C+Lis)
26. Guodong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. [Exploring various knowledge in relation extraction](https://md.research.elamur.ai/free/en/papers/zhou-2005-exploring-various-knowledge-relation-extraction/). In ACL-05, pages 427–434, Ann Arbor, MI.
27. Guodong Zhou, Min Zhang, Donghong Ji, and Qiaoming Zhu. 2007. Tree kernel-based relation extraction with context-sensitive structured parse tree information. In EMNLP/CoNLL 2007. [find](https://scholar.google.com/scholar?q=Guodong+Zhou%2C+Min+Zhang%2C+Donghong+Ji%2C+and+Qiaoming+Zhu%2E+2007%2E+Tree+kernel%2Dbased+relation+extraction+with+context%2Dsensitive+structured+parse+tree+information%2E+In+EMNLP%2FCoNLL+2007%2E)

*Updated 29 October 2026*