--- title: "Automatic Retrieval and Clustering of Similar — Lin 1998" description: "The paper addresses how to discover semantically similar words automatically from raw text, as a first step toward learning word meaning and building…" authors: ["Dekang Lin"] year: 1998 venue: "36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2" source: "https://aclanthology.org/P98-2127/" doi: "10.3115/980691.980696" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lin-1998-automatic-retrieval-clustering-similar-words/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-1998-automatic-retrieval-clustering-similar-words/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1998](https://md.research.elamur.ai/free/en/papers/1998/) / Automatic Retrieval and Clustering of Similar Words # Automatic Retrieval and Clustering of Similar Words **Authors:** [Dekang Lin](https://md.research.elamur.ai/free/en/authors/dekang-lin/) · **Year:** 1998 · **Venue:** 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2 · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P98-2127/) · [PDF](https://aclanthology.org/P98-2127.pdf) · [DOI 10.3115/980691.980696](https://doi.org/10.3115/980691.980696) · [OpenAlex](https://openalex.org/W2166776180) **Citations:** 1545 citations (openalex-snapshot · 2026-10-14) · [515](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper addresses how to discover semantically similar words automatically from raw text, as a first step toward learning word meaning and building corpus-specific thesauri. Lin parses a 64-million-word newswire corpus with a broad-coverage dependency parser and represents each word by the grammatical contexts it appears in, stored as dependency triples. He then defines an information-theoretic similarity measure: two words are similar to the extent that the information carried by their shared contexts makes up a large part of the total information in both descriptions. Applying this measure to frequent nouns, verbs and adjectives/adverbs yields a thesaurus that lists the 200 nearest neighbours for each word. The paper's main methodological contribution is a direct, objective evaluation. Both WordNet and Roget are converted into the same ranked-neighbour format, and entries for the same word are compared across resources. The automatic thesaurus agrees with WordNet more closely than Roget does. This shows that distributional thesauri can rival hand-built resources and can be measured against them quantitatively. ## Key points - **Corpus:** 64M words in total, drawn from Wall Street Journal (24M), San Jose Mercury (21M) and AP Newswire (19M). Parsing produced 56.5M dependency triples, of which 8.7M are unique. - **Vocabulary covered:** all words occurring at least 100 times, namely 5469 nouns, 2173 verbs and 2632 adjectives/adverbs. Each word's thesaurus entry keeps its top 200 neighbours. - **Mutual reciprocity:** the program found 543 noun pairs, 212 verb pairs and 382 adjective/adverb pairs that are respective nearest neighbours, meaning each word is the other's top match. - **Evaluation set:** 4294 nouns present in WordNet 1.5 and in Roget. - Average similarity to WordNet entries was 0.212 for the proposed measure, compared with 0.178 for Roget. - Competing measures scored lower: Hindle_r 0.204, cosine 0.199 and Hindle 0.165. - **Significance:** paired per-entry differences have 95% confidence intervals entirely above zero. This supports the ranking sim > Hindle_r > cosine. The comparison also shows that using all dependency types, rather than only subject/object relations, clearly helps. ## Interesting ideas - **Common evaluation format.** Hand-built resources are converted into the same "word → weighted neighbour list" format as the automatic output. This allows heterogeneous resources to be compared on equal footing, and the trick could be reused to benchmark distributional models for Ukrainian against existing lexical databases. - **Information-weighted overlap.** Shared contexts are weighted by their information content (pointwise mutual information), and contexts with non-positive association are discarded. This is a simple way to keep frequent, uninformative contexts from dominating the similarity score. - **Rich syntactic context features.** Word context is described through typed dependency relations, including inverse relations such as "obj-of", rather than through bag-of-words windows. The gain over a subject/object-only variant suggests that richer syntactic features pay off, which is relevant whenever a dependency parser (e.g. a UD-trained one) is available. - **Paired significance testing.** Systems are compared through per-item differences and the standard deviation of the mean, rather than through raw averages alone. This is a cheap way to make claims about small score gaps statistically defensible. - **Similarity trees for word senses.** Each neighbour is attached under whichever already-placed word it most resembles. The resulting branches roughly separate a word's senses, which makes this a lightweight heuristic for exploring polysemy without a sense inventory. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 515 papers of our library cite it (of the 140884 whose bibliographies we read); 44 of them have a page here, 471 not yet. - [Evaluating WordNet-based Measures of Lexical Semantic Relatedness](https://md.research.elamur.ai/free/en/papers/budanitsky-2006-evaluating-wordnet-based-measures-lexical/) - [Semantic Compositionality through Recursive Matrix-Vector Spaces](https://md.research.elamur.ai/free/en/papers/socher-2012-semantic-compositionality-through-recursive-matrix/) - [A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches](https://md.research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/) - [Dependency-Based Construction of Semantic Space Models](https://md.research.elamur.ai/free/en/papers/pado-2007-dependency-based-construction-semantic-space/) - [Distributional Memory: A General Framework for Corpus-Based Semantics](https://md.research.elamur.ai/free/en/papers/baroni-2010-distributional-memory-general-framework-corpus/) - [Learning Subjective Language](https://md.research.elamur.ai/free/en/papers/wiebe-2004-learning-subjective-language/) - [Automatic Evaluation of Topic Coherence](https://md.research.elamur.ai/free/en/papers/newman-2010-automatic-evaluation-topic-coherence/) - [Linguistic Regularities in Sparse and Explicit Word Representations](https://md.research.elamur.ai/free/en/papers/levy-2014-linguistic-regularities-sparse-explicit-word/) - [Extracting Paraphrases from a Parallel Corpus](https://md.research.elamur.ai/free/en/papers/barzilay-2001-extracting-paraphrases-parallel-corpus/) - [Automatic Identification of Word Translations from Unrelated English and German Corpora](https://md.research.elamur.ai/free/en/papers/rapp-1999-automatic-identification-word-translations-unrelated/) - [Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment](https://md.research.elamur.ai/free/en/papers/barzilay-2003-learning-paraphrase-unsupervised-approach-multiple/) - [A Structured Vector Space Model for Word Meaning in Context](https://md.research.elamur.ai/free/en/papers/erk-2008-structured-vector-space-model-word/) - [Shallow Semantic Parsing using Support Vector Machines](https://md.research.elamur.ai/free/en/papers/pradhan-2004-shallow-semantic-parsing-support-vector/) - [Finding Predominant Word Senses in Untagged Text](https://md.research.elamur.ai/free/en/papers/onto-2004-mccarthy-finding-predominant-word-senses-untagged/) - [Generating Phrasal and Sentential Paraphrases: A Survey of Data-Driven Methods](https://md.research.elamur.ai/free/en/papers/madnani-2010-generating-phrasal-sentential-paraphrases-survey/) - [Frame-Semantic Parsing](https://md.research.elamur.ai/free/en/papers/das-2014-frame-semantic-parsing/) - [UMBC_EBIQUITY-CORE: Semantic Textual Similarity Systems](https://md.research.elamur.ai/free/en/papers/han-2013-umbc-ebiquity-core-semantic-textual/) - [Do Supervised Distributional Methods Really Learn Lexical Inference Relations?](https://md.research.elamur.ai/free/en/papers/levy-2015-do-supervised-distributional-methods-really/) - [An Empirical Model of Multiword Expression Decomposability](https://md.research.elamur.ai/free/en/papers/baldwin-2003-empirical-model-multiword-expression-decomposability/) - [Unsupervised Type and Token Identification of Idiomatic Expressions](https://md.research.elamur.ai/free/en/papers/fazly-2009-unsupervised-type-token-identification-idiomatic/) - [The Distributional Inclusion Hypotheses and Lexical Entailment](https://md.research.elamur.ai/free/en/papers/geffet-2005-distributional-inclusion-hypotheses-lexical-entailment/) - [A Survey of Paraphrasing and Textual Entailment Methods](https://md.research.elamur.ai/free/en/papers/androutsopoulos-2009-survey-paraphrasing-textual-entailment-methods/) - [Similarity of Semantic Relations](https://md.research.elamur.ai/free/en/papers/turney-2006-similarity-semantic-relations/) - [Joint Entity and Event Coreference Resolution across Documents](https://md.research.elamur.ai/free/en/papers/lee-2012-joint-entity-event-coreference-resolution/) - [Learning Discriminative Projections for Text Similarity Measures](https://md.research.elamur.ai/free/en/papers/yih-2011-learning-discriminative-projections-text-similarity/) - [Semantic Class Learning from the Web with Hyponym Pattern Linkage Graphs](https://md.research.elamur.ai/free/en/papers/kozareva-2008-semantic-class-learning-web-hyponym/) - [Metaphor Identification Using Verb and Noun Clustering](https://md.research.elamur.ai/free/en/papers/shutova-2010-metaphor-identification-verb-noun-clustering/) - [Learning to Distinguish Hypernyms and Co-Hyponyms](https://md.research.elamur.ai/free/en/papers/weeds-2014-learning-distinguish-hypernyms-co-hyponyms/) - [Text: now in 2D! A framework for lexical expansion with contextual similarity](https://md.research.elamur.ai/free/en/papers/biemann-2013-text-now-2d-framework-lexical/) - [Unsupervised Recognition of Literal and Non-Literal Use of Idiomatic Expressions](https://md.research.elamur.ai/free/en/papers/sporleder-2009-unsupervised-recognition-literal-non-literal/) - [Towards Robust Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/pradhan-2008-robust-semantic-role-labeling/) - [Automatically Constructing a Lexicon of Verb Phrase Idiomatic Combinations](https://md.research.elamur.ai/free/en/papers/fazly-2006-automatically-constructing-lexicon-verb-phrase/) - [Lexical Chaining for Measuring Discourse Coherence Quality in Test-taker Essays](https://md.research.elamur.ai/free/en/papers/somasundaran-2014-lexical-chaining-measuring-discourse-coherence/) - [An Unsupervised Approach to Prepositional Phrase Attachment using Contextually Similar Words](https://md.research.elamur.ai/free/en/papers/aclbulk-2000-pantel-unsupervised-approach-prepositional-phrase-attachment/) - [Semantic Class Induction and Coreference Resolution](https://md.research.elamur.ai/free/en/papers/ng-2007-semantic-class-induction-coreference-resolution/) - [Semi-Supervised Frame-Semantic Parsing for Unknown Predicates](https://md.research.elamur.ai/free/en/papers/das-2011-semi-supervised-frame-semantic-parsing/) - [Semantic Inference at the Lexical-Syntactic Level for Textual Entailment Recognition](https://md.research.elamur.ai/free/en/papers/haim-2007-semantic-inference-lexical-syntactic-level/) - [Antecedent Selection Techniques for High-Recall Coreference Resolution](https://md.research.elamur.ai/free/en/papers/versley-2007-antecedent-selection-techniques-high-recall/) - [Classification of Discourse Coherence Relations: An Exploratory Study using Multiple Knowledge Sources](https://md.research.elamur.ai/free/en/papers/wellner-2006-classification-discourse-coherence-relations-exploratory/) - [Multi-word expressions in textual inference: Much ado about nothing?](https://md.research.elamur.ai/free/en/papers/aclbulk-2009-marneffe-multi-word-expressions-textual-inference/) - [Automatic Identification of Infrequent Word Senses](https://md.research.elamur.ai/free/en/papers/aclbulk-2004-mccarthy-automatic-identification-infrequent-word-senses/) - [Word Sense Induction with Attentive Context Clustering](https://md.research.elamur.ai/free/en/papers/aclbulk-2021-stekel-word-sense-induction-attentive-context/) - [Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic](https://md.research.elamur.ai/free/en/papers/aclbulk-2009-dinur-unsupervised-concept-discovery-hebrew-simple/) - [Automatic Thesaurus Construction for Modern Hebrew](https://md.research.elamur.ai/free/en/papers/lingbulk-2018-liebeskind-automatic-thesaurus-construction-modern-hebrew/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lin1998automatic, title = {{Automatic Retrieval and Clustering of Similar Words}}, author = {Lin, Dekang}, booktitle = {36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2}, year = {1998}, doi = {10.3115/980691.980696}, url = {https://doi.org/10.3115/980691.980696} } ``` - [← previous The Berkeley FrameNet Project](https://md.research.elamur.ai/free/en/papers/lex-1998-baker-berkeley-framenet/) - [next → A Language Modeling Approach to Information Retrieval](https://md.research.elamur.ai/free/en/papers/ponte-1998-language-modeling-approach-information/) - [↑ back to papers of 1998](https://md.research.elamur.ai/free/en/papers/1998/) ## References 18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Hiyan Alshawi and David Carter. 1994. Training and scaling preference functions for disambiguation. Computational Linguistics, 20(4):635-648, December. [find](https://scholar.google.com/scholar?q=Hiyan+Alshawi+and+David+Carter%2E+1994%2E+Training+and+scaling+preference+functions+for+disambiguation%2E+Computational+Linguistics%2C+20%284%29%3A635%2D648%2C+December%2E) 2. Ido Dagan, Shaul Marcus, and Shaul Markovitch. 1993. Contextual word similarity and estimation from sparse data. In Proceedings of ACL-93, pages 164-171, Columbus, Ohio, June. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Shaul+Marcus%2C+and+Shaul+Markovitch%2E+1993%2E+Contextual+word+similarity+and+estimation+from+sparse+data%2E+In+Proceedings+of+ACL%2D93%2C+pages+164%2D171%2C+Columbus%2C+Ohio%2C+June%2E) 3. Ido Dagan, Fernando Pereira, and Lillian Lee. 1994. Similarity-based estimation of word cooccurrence probabilities. In Proceedings of the 32nd Annual Meeting of the ACL, pages 272-278, Las Cruces, NM. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Fernando+Pereira%2C+and+Lillian+Lee%2E+1994%2E+Similarity%2Dbased+estimation+of+word+cooccurrence+probabilities%2E+In+Proceedings+of+the+32nd+Annual+Meeting+of+the+ACL%2C+pages+272%2D278%2C+Las+Cruces%2C+NM%2E) 4. Ido Dagan, Lillian Lee, and Fernando Pereira. 1997. Similarity-based method for word sense disambiguation. In Proceedings of the 35th Annual Meeting of the ACL, pages 56-63, Madrid, Spain. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Lillian+Lee%2C+and+Fernando+Pereira%2E+1997%2E+Similarity%2Dbased+method+for+word+sense+disambiguation%2E+In+Proceedings+of+the+35th+Annual+Meeting+of+the+ACL%2C+pages+56%2D63%2C+Madrid%2C+Spain%2E) 5. Ute Essen and Volker Steinbiss. 1992. Cooccurrence smoothing for stochastic language modeling. In Proceedings oflCASSP, volume 1, pages 161-164. [find](https://scholar.google.com/scholar?q=Ute+Essen+and+Volker+Steinbiss%2E+1992%2E+Cooccurrence+smoothing+for+stochastic+language+modeling%2E+In+Proceedings+oflCASSP%2C+volume+1%2C+pages+161%2D164%2E) 6. W. B. Frakes and R. Baeza-Yates, editors. 1992. In. formation Retrieval, Data Structure and Algorithms. [find](https://scholar.google.com/scholar?q=W%2E+B%2E+Frakes+and+R%2E+Baeza%2DYates%2C+editors%2E+1992%2E+In%2E+formation+Retrieval%2C+Data+Structure+and+Algorithms%2E) 7. D. Gentner. 1982. Why nouns are learned before verbs: Linguistic relativity versus natural partitioning. In S. A. Kuczaj, editor, Language development: Vol. 2. Language, thought, and culture, pages 301-334. Erlbaum, Hillsdale, NJ. [find](https://scholar.google.com/scholar?q=D%2E+Gentner%2E+1982%2E+Why+nouns+are+learned+before+verbs%3A+Linguistic+relativity+versus+natural+partitioning%2E+In+S%2E+A%2E+Kuczaj%2C+editor%2C+Language+development%3A+Vol%2E+2%2E+Language%2C+thought%2C+and+culture%2C+pages+30) 8. Gregory Grefenstette. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer Academic Press, Boston, MA. [find](https://scholar.google.com/scholar?q=Gregory+Grefenstette%2E+1994%2E+Explorations+in+Automatic+Thesaurus+Discovery%2E+Kluwer+Academic+Press%2C+Boston%2C+MA%2E) 9. Donald Hindle. 1990. Noun classification from predicate-argument structures. In Proceedings of ACL-90, pages 268-275, Pittsburg, Pennsylvania, June. [find](https://scholar.google.com/scholar?q=Donald+Hindle%2E+1990%2E+Noun+classification+from+predicate%2Dargument+structures%2E+In+Proceedings+of+ACL%2D90%2C+pages+268%2D275%2C+Pittsburg%2C+Pennsylvania%2C+June%2E) [our card](https://md.research.elamur.ai/free/en/papers/hindle-1990-noun-classification-predicate-argument-structures/) 10. Dekang Lin. 1993. Principle-based parsing without overgeneration. In Proceedings of ACL-93, pages 112-120, Columbus, Ohio. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1993%2E+Principle%2Dbased+parsing+without+overgeneration%2E+In+Proceedings+of+ACL%2D93%2C+pages+112%2D120%2C+Columbus%2C+Ohio%2E) 11. Dekang Lin. 1994. Principarman efficient, broad-coverage, principle-based parser. In Proceedings of COLING-94, pages 482-488. Kyoto, Japan. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1994%2E+Principarman+efficient%2C+broad%2Dcoverage%2C+principle%2Dbased+parser%2E+In+Proceedings+of+COLING%2D94%2C+pages+482%2D488%2E+Kyoto%2C+Japan%2E) 12. Dekang Lin. 1997. Using syntactic dependency as local context to resolve word sense ambiguity. In Proceedings of ACL/EACL-97, pages 64-71, Madrid, Spain, July. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1997%2E+Using+syntactic+dependency+as+local+context+to+resolve+word+sense+ambiguity%2E+In+Proceedings+of+ACL%2FEACL%2D97%2C+pages+64%2D71%2C+Madrid%2C+Spain%2C+July%2E) 13. George A. Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J. Miller. 1990. Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 3(4):235-244. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller%2C+Richard+Beckwith%2C+Christiane+Fellbaum%2C+Derek+Gross%2C+and+Katherine+J%2E+Miller%2E+1990%2E+Introduction+to+WordNet%3A+An+on%2Dline+lexical+database%2E+International+Journal+of+Lexicography%2C+3%284%29%3A2) 14. George A. Miller. 1990. WordNet: An on-line lexical database. International Journal of Lexicography, 3(4):235-312. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller%2E+1990%2E+WordNet%3A+An+on%2Dline+lexical+database%2E+International+Journal+of+Lexicography%2C+3%284%29%3A235%2D312%2E) 15. Eugene A. Nida. 1975. ComponentialAnalysis of Meaning. The Hague, Mouton. [find](https://scholar.google.com/scholar?q=Eugene+A%2E+Nida%2E+1975%2E+ComponentialAnalysis+of+Meaning%2E+The+Hague%2C+Mouton%2E) 16. F. Pereira, N. Tishby, and L. Lee. 1993. Distributional Clustering of English Words. In Proceedings of ACL-93, pages 183-190, Ohio State University, Columbus, Ohio. [find](https://scholar.google.com/scholar?q=F%2E+Pereira%2C+N%2E+Tishby%2C+and+L%2E+Lee%2E+1993%2E+Distributional+Clustering+of+English+Words%2E+In+Proceedings+of+ACL%2D93%2C+pages+183%2D190%2C+Ohio+State+University%2C+Columbus%2C+Ohio%2E) [our card](https://md.research.elamur.ai/free/en/papers/pereira-1993-distributional-clustering-english-words/) 17. Gerda Ruge. 1992. Experiments on linguistically based term associations. Information Processing & Management, 28(3):317-332. [find](https://scholar.google.com/scholar?q=Gerda+Ruge%2E+1992%2E+Experiments+on+linguistically+based+term+associations%2E+Information+Processing+%26+Management%2C+28%283%29%3A317%2D332%2E) 18. Frank Smadja. 1993. Retrieving collocations from text: Xtract. Computational Linguistics, 19(1): 143-178. [find](https://scholar.google.com/scholar?q=Frank+Smadja%2E+1993%2E+Retrieving+collocations+from+text%3A+Xtract%2E+Computational+Linguistics%2C+19%281%29%3A+143%2D178%2E) [our card](https://md.research.elamur.ai/free/en/papers/smadja-1993-retrieving-collocations-text-xtract/)