---
title: "Semantic Class Learning from the Web with Hyponym Pattern"
description: "This unchecked loop (\"reckless bootstrapping\") quickly drifts off-topic, so the authors build… 92% recall. 28 references, cited by 82 papers here."
authors: ["Zornitsa Kozareva", "Ellen Riloff", "Eduard Hovy"]
year: 2008
venue: "Proceedings of ACL-08: HLT"
source: "https://aclanthology.org/P08-1119/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kozareva-2008-semantic-class-learning-web-hyponym/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kozareva-2008-semantic-class-learning-web-hyponym/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2008](https://md.research.elamur.ai/free/en/papers/2008/) / Semantic Class Learning from the Web with Hyponym Pattern Linkage Graphs

# Semantic Class Learning from the Web with Hyponym Pattern Linkage Graphs

**Authors:** [Zornitsa Kozareva](https://md.research.elamur.ai/free/en/authors/zornitsa-kozareva/), [Ellen Riloff](https://md.research.elamur.ai/free/en/authors/ellen-riloff/) and [Eduard Hovy](https://md.research.elamur.ai/free/en/authors/eduard-hovy/) · **Year:** 2008 · **Venue:** [Proceedings of ACL-08: HLT](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2008](https://md.research.elamur.ai/free/en/venues/acl/2008/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P08-1119/) · [PDF](https://aclanthology.org/P08-1119.pdf) · [OpenAlex](https://openalex.org/W2145328028)

**Citations:** 191 citations (OpenAlex, 2026-10-14) · [82](https://md.research.elamur.ai/free/en/papers/kozareva-2008-semantic-class-learning-web-hyponym/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles automatic construction of semantic lexicons, meaning lists of words that belong to a given category, using the web with almost no supervision. The only input is a class name and one seed member. Candidates are harvested with a single doubly-anchored Hearst-style pattern, "\<class name> such as \<class member> and *", which is issued as a web query. Every newly extracted word is fed back as a new seed. This unchecked loop ("reckless bootstrapping") quickly drifts off-topic, so the authors build hyponym pattern linkage graphs (HPLGs) that record which instances discover which others. From these graphs they derive two signals: popularity (how often a word is found by others) and productivity (how often it finds new words). They propose two algorithms: one uses popularity to guide bootstrapping on the fly, and one runs an exhaustive search first and then ranks candidates by graph properties. Tested on U.S. states, countries, singers and fish, the methods achieve high accuracy, and the productivity-based ranking works best. No tagging, parsing or labelled data is needed, so the approach can be applied to arbitrary categories.

## Key points
- The authors evaluate on four classes: U.S. states (gold list of 50), countries (gold list of 194), and two large open classes, singers and fish.
- Reckless bootstrapping loses accuracy fast as iterations pile up. Countries fall from .80 to .16 over four iterations, and states fall from .79 to .18 by the third. Singers hold up better (.91 to .83).
- The best system is the two-step method: exhaustive search, then ranking by the Out-degree (productivity) score. Words with zero productivity are cut from the list. In the singer run seeded with Placido Domingo, 400 candidates were produced and the list was cut at 180.
- Results are stable across seeds. All five random country seeds (e.g., Guinea-Bissau, Uganda) reached at least 92% recall with at least 90% precision.
- Errors fall into five types:
  - faulty name extraction,
  - former or borderline members,
  - spelling and name variants,
  - factually wrong web text,
  - broken tokens in snippets.

## Interesting ideas
- **Double anchoring for disambiguation.** Putting both the category and a known member into the query makes each one narrow the meaning of the other. This is a cheap way to steer extraction toward the intended sense of ambiguous words, and it could carry over to other languages or to search-based data collection.
- **Popularity plus productivity as a check.** A candidate is trusted when other members lead to it and it also leads to other members. This kind of two-directional consistency in a discovery graph could filter noise in any bootstrapped extraction pipeline.
- **Zero productivity as a stopping rule.** Truncating the ranked list where candidates stop yielding new instances gives a cutoff without any labelled data. It is useful when no development set exists.
- **Rank overflow reveals class definition problems.** The entries ranked just below the true states were former Soviet republics, which showed that the class name itself was underspecified. Inspecting the borderline region of a ranking is a handy diagnostic for how a category was defined.
- **Not every error is harmful.** The error analysis notes that former members and name variants may actually be useful in a dictionary or ontology. Evaluation should therefore consider what the downstream resource needs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 104 works · 33,876 citations
- [University of Utah](https://md.research.elamur.ai/free/en/universities/university-of-utah/) — University, Salt Lake City, United States · 29 works · 5,781 citations
- [University of Alicante](https://md.research.elamur.ai/free/en/universities/university-of-alicante/) — University, Alicante, Spain · 14 works · 1,081 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

## Appears in

Agglomerations: [🏙️ Salt Lake City](https://md.research.elamur.ai/free/en/agglomerations/salt-lake-city/)

## Cite

Cite as: Zornitsa Kozareva, Ellen Riloff and Eduard Hovy. 2008. Semantic Class Learning from the Web with Hyponym Pattern Linkage Graphs. Proceedings of ACL-08: HLT.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kozareva2008semantic,
  title = {{Semantic Class Learning from the Web with Hyponym Pattern Linkage Graphs}},
  author = {Zornitsa Kozareva, Ellen Riloff and Eduard Hovy},
  booktitle = {Proceedings of ACL-08: HLT},
  year = {2008},
  url = {https://aclanthology.org/P08-1119/}
}
```

- [← previous Feature Forest Models for Probabilistic HPSG Parsing](https://md.research.elamur.ai/free/en/papers/miyao-2008-feature-forest-models-probabilistic-hpsg/)
- [next → Joint Unsupervised Coreference Resolution with Markov Logic](https://md.research.elamur.ai/free/en/papers/poon-2008-joint-unsupervised-coreference-resolution-markov/)
- [↑ back to papers of 2008](https://md.research.elamur.ai/free/en/papers/2008/)

## References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Berland and E. Charniak. 1999. [Finding Parts in Very Large Corpora](https://md.research.elamur.ai/free/en/papers/berland-1999-finding-parts-very-large-corpora/). In Proc. of the 37th Annual Meeting of the Association for Computational Linguistics.
2. S. Borgatti and M. Everett. 2006. A graph-theoretic perspective on centrality. Social Networks, 28(4). [find](https://scholar.google.com/scholar?q=S%2E+Borgatti+and+M%2E+Everett%2E+2006%2E+A+graph%2Dtheoretic+perspective+on+centrality%2E+Social+Networks%2C+28%284%29%2E)
3. S. Caraballo. 1999. Automatic Acquisition of a Hypernym-Labeled Noun Hierarchy from Text. In Proc. of the 37th Annual Meeting of the Association for Computational Linguistics, pages 120–126. [find](https://scholar.google.com/scholar?q=S%2E+Caraballo%2E+1999%2E+Automatic+Acquisition+of+a+Hypernym%2DLabeled+Noun+Hierarchy+from+Text%2E+In+Proc%2E+of+the+37th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+120%E2%80%93126%2E)
4. P. Cimiano and J. Volker. 2005. Towards large-scale, open-domain and ontology-based named entity classification. In Proc. of Recent Advances in Natural Language Processing, pages 166–172. [find](https://scholar.google.com/scholar?q=P%2E+Cimiano+and+J%2E+Volker%2E+2005%2E+Towards+large%2Dscale%2C+open%2Ddomain+and+ontology%2Dbased+named+entity+classification%2E+In+Proc%2E+of+Recent+Advances+in+Natural+Language+Processing%2C+pages+166%E2%80%93172%2E)
5. D. Davidov and A. Rappoport. 2006. Efficient unsupervised discovery of word categories using symmetric patterns and high frequency words. In Proc. of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL. [find](https://scholar.google.com/scholar?q=D%2E+Davidov+and+A%2E+Rappoport%2E+2006%2E+Efficient+unsupervised+discovery+of+word+categories+using+symmetric+patterns+and+high+frequency+words%2E+In+Proc%2E+of+the+21st+International+Conference+on+Computational)
6. O. Etzioni, M. Cafarella, D. Downey, A. Popescu, T. Shaked, S. Soderland, D. Weld, and A. Yates. 2005. Unsupervised named-entity extraction from the web: an experimental study. Artificial Intelligence, 165(1):91–134, June. [find](https://scholar.google.com/scholar?q=O%2E+Etzioni%2C+M%2E+Cafarella%2C+D%2E+Downey%2C+A%2E+Popescu%2C+T%2E+Shaked%2C+S%2E+Soderland%2C+D%2E+Weld%2C+and+A%2E+Yates%2E+2005%2E+Unsupervised+named%2Dentity+extraction+from+the+web%3A+an+experimental+study%2E+Artificial+Intelligence)
7. M.B. Fleischman and E.H. Hovy. 2002. Fine grained classification of named entities. In Proc. of the 19th International Conference on Computational Linguistics, pages 1–7. [find](https://scholar.google.com/scholar?q=M%2EB%2E+Fleischman+and+E%2EH%2E+Hovy%2E+2002%2E+Fine+grained+classification+of+named+entities%2E+In+Proc%2E+of+the+19th+International+Conference+on+Computational+Linguistics%2C+pages+1%E2%80%937%2E)
8. C. Freeman. 1979. Centrality in social networks: Conceptual clarification. Social Networks, 1:215–239. [find](https://scholar.google.com/scholar?q=C%2E+Freeman%2E+1979%2E+Centrality+in+social+networks%3A+Conceptual+clarification%2E+Social+Networks%2C+1%3A215%E2%80%93239%2E)
9. R. Girju, A. Badulescu, and D. Moldovan. 2003. [Learning semantic constraints for the automatic discovery of part-whole relations](https://md.research.elamur.ai/free/en/papers/girju-2003-learning-semantic-constraints-automatic-discovery/). In Proc. of Conference of HLT / North American Chapter of the Association for Computational Linguistics.
10. M. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proc. of the 14th conference on Computational linguistics, pages 539–545. [find](https://scholar.google.com/scholar?q=M%2E+Hearst%2E+1992%2E+Automatic+acquisition+of+hyponyms+from+large+text+corpora%2E+In+Proc%2E+of+the+14th+conference+on+Computational+linguistics%2C+pages+539%E2%80%93545%2E)
11. D. Lin and P. Pantel. 2002. Concept discovery from text. In Proc. of the 19th International Conference on Computational linguistics, pages 1–7. [find](https://scholar.google.com/scholar?q=D%2E+Lin+and+P%2E+Pantel%2E+2002%2E+Concept+discovery+from+text%2E+In+Proc%2E+of+the+19th+International+Conference+on+Computational+linguistics%2C+pages+1%E2%80%937%2E)
12. D. Lin. 1998. [Automatic retrieval and clustering of similar words](https://md.research.elamur.ai/free/en/papers/lin-1998-automatic-retrieval-clustering-similar-words/). In Proc. of the 17th international conference on Computational linguistics, pages 768–774.
13. G. Mann. 2002. Fine-grained proper noun ontologies for question answering. In Proc. of the 19th International Conference on Computational Linguistics, pages 1–7. [find](https://scholar.google.com/scholar?q=G%2E+Mann%2E+2002%2E+Fine%2Dgrained+proper+noun+ontologies+for+question+answering%2E+In+Proc%2E+of+the+19th+International+Conference+on+Computational+Linguistics%2C+pages+1%E2%80%937%2E)
14. G. Miller. 1990. Wordnet: An On-line Lexical Database. International Journal of Lexicography, 3(4). [find](https://scholar.google.com/scholar?q=G%2E+Miller%2E+1990%2E+Wordnet%3A+An+On%2Dline+Lexical+Database%2E+International+Journal+of+Lexicography%2C+3%284%29%2E)
15. R. Navigli and M. Lapata. 2007. [Graph connectivity measures for unsupervised word sense disambiguation](https://md.research.elamur.ai/free/en/papers/navigli-2007-graph-connectivity-measures-unsupervised-word/). In Proc. of the 20th International Joint Conference on Artificial Intelligence, pages 1683–1688.
16. M. Paşca. 2004. Acquisition of categorized named entities for web search. In Proc. of the Thirteenth ACM International Conference on Information and Knowledge Management, pages 137–145. [find](https://scholar.google.com/scholar?q=M%2E+Pas%CC%A7ca%2E+2004%2E+Acquisition+of+categorized+named+entities+for+web+search%2E+In+Proc%2E+of+the+Thirteenth+ACM+International+Conference+on+Information+and+Knowledge+Management%2C+pages+137%E2%80%93145%2E)
17. M. Paşca. 2007a. Organizing and searching the world wide web of facts – step two: harnessing the wisdom of the crowds. In Proc. of the 16th International Conference on World Wide Web, pages 101–110. [find](https://scholar.google.com/scholar?q=M%2E+Pas%CC%A7ca%2E+2007a%2E+Organizing+and+searching+the+world+wide+web+of+facts+%E2%80%93+step+two%3A+harnessing+the+wisdom+of+the+crowds%2E+In+Proc%2E+of+the+16th+International+Conference+on+World+Wide+Web%2C+pages+101%E2%80%93110%2E)
18. M. Paşca. 2007b. Weakly-supervised discovery of named entities using web search queries. In Proc. of the sixteenth ACM conference on Conference on information and knowledge management, pages 683–690. [find](https://scholar.google.com/scholar?q=M%2E+Pas%CC%A7ca%2E+2007b%2E+Weakly%2Dsupervised+discovery+of+named+entities+using+web+search+queries%2E+In+Proc%2E+of+the+sixteenth+ACM+conference+on+Conference+on+information+and+knowledge+management%2C+pages+683%E2%80%93690%2E)
19. L. Page, S. Brin, R. Motwani, and T. Winograd. 1998. [The pagerank citation ranking: Bringing order to the web](https://md.research.elamur.ai/free/en/papers/page-1999-pagerank-citation-ranking-bringing/). Technical report, Stanford Digital Library Technologies Project.
20. P. Pantel and D. Ravichandran. 2004. [Automatically labeling semantic classes](https://md.research.elamur.ai/free/en/papers/pantel-2004-automatically-labeling-semantic-classes/). In Proc. of Conference of HLT / North American Chapter of the Association for Computational Linguistics, pages 321–328.
21. P. Pantel, D. Ravichandran, and E. Hovy. 2004. Towards terascale knowledge acquisition. In Proc. of the 20th international conference on Computational Linguistics, page 771. [find](https://scholar.google.com/scholar?q=P%2E+Pantel%2C+D%2E+Ravichandran%2C+and+E%2E+Hovy%2E+2004%2E+Towards+terascale+knowledge+acquisition%2E+In+Proc%2E+of+the+20th+international+conference+on+Computational+Linguistics%2C+page+771%2E)
22. W. Phillips and E. Riloff. 2002. Exploiting Strong Syntactic Heuristics and Co-Training to Learn Semantic Lexicons. In Proc. of the 2002 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=W%2E+Phillips+and+E%2E+Riloff%2E+2002%2E+Exploiting+Strong+Syntactic+Heuristics+and+Co%2DTraining+to+Learn+Semantic+Lexicons%2E+In+Proc%2E+of+the+2002+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2E)
23. E. Riloff and R. Jones. 1999. Learning Dictionaries for Information Extraction by Multi-Level Bootstrapping. In Proc. of the Sixteenth National Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=E%2E+Riloff+and+R%2E+Jones%2E+1999%2E+Learning+Dictionaries+for+Information+Extraction+by+Multi%2DLevel+Bootstrapping%2E+In+Proc%2E+of+the+Sixteenth+National+Conference+on+Artificial+Intelligence%2E)
24. E. Riloff and J. Shepherd. 1997. [A Corpus-Based Approach for Building Semantic Lexicons](https://md.research.elamur.ai/free/en/papers/riloff-1997-corpus-based-approach-building-semantic/). In Proc. of the Second Conference on Empirical Methods in Natural Language Processing, pages 117–124.
25. B. Roark and E. Charniak. 1998. Noun-phrase Co-occurrence Statistics for Semi-automatic Semantic Lexicon Construction. In Proc. of the 36th Annual Meeting of the Association for Computational Linguistics, pages 1110–1116. [find](https://scholar.google.com/scholar?q=B%2E+Roark+and+E%2E+Charniak%2E+1998%2E+Noun%2Dphrase+Co%2Doccurrence+Statistics+for+Semi%2Dautomatic+Semantic+Lexicon+Construction%2E+In+Proc%2E+of+the+36th+Annual+Meeting+of+the+Association+for+Computational+Linguist)
26. H. Tanev and B. Magnini. 2006. Weakly supervised approaches for ontology population. In Proc. of 11st Conference of the European Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=H%2E+Tanev+and+B%2E+Magnini%2E+2006%2E+Weakly+supervised+approaches+for+ontology+population%2E+In+Proc%2E+of+11st+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%2E)
27. M. Thelen and E. Riloff. 2002. [A Bootstrapping Method for Learning Semantic Lexicons Using Extraction Pattern Contexts](https://md.research.elamur.ai/free/en/papers/thelen-2002-bootstrapping-method-learning-semantic-lexicons/). In Proc. of the 2002 Conference on Empirical Methods in Natural Language Processing.
28. D. Widdows and B. Dorow. 2002. [A graph model for unsupervised lexical acquisition](https://md.research.elamur.ai/free/en/papers/widdows-2002-graph-model-unsupervised-lexical-acquisition/). In Proc. of the 19th International Conference on Computational Linguistics, pages 1–7.

*Updated 24 October 2026*