---
title: "Measures of Distributional Similarity — Lee 1999"
description: "The work offers both a better metric and a principled way to classify similarity functions. 51% error rate. 33 references, cited by 71 papers here."
authors: ["Lillian Lee"]
year: 1999
venue: "Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P99-1004/"
doi: "10.3115/1034678.1034693"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lee-1999-measures-distributional-similarity/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-1999-measures-distributional-similarity/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1999](https://md.research.elamur.ai/free/en/papers/1999/) / Measures of Distributional Similarity

# Measures of Distributional Similarity

**Authors:** [Lillian Lee](https://md.research.elamur.ai/free/en/authors/lillian-lee/) · **Year:** 1999 · **Venue:** Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P99-1004/) · [PDF](https://aclanthology.org/P99-1004.pdf) · [DOI 10.3115/1034678.1034693](https://doi.org/10.3115/1034678.1034693) · [OpenAlex](https://openalex.org/W2120084270)

**Citations:** 583 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/lee-1999-measures-distributional-similarity/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper tackles sparse data in statistical NLP: how to estimate the probability of a word pair that never appeared in training. Lee uses distance-weighted averaging, which builds the estimate from the cooccurrence behaviour of distributionally similar words. She asks which similarity function best picks those proxy words. The testbed is verb–object pairs from the 1988 Associated Press newswire, restricted to the 1000 most frequent nouns, with a pseudoword-style binary decision task on unseen pairs.

Seven measures are compared using a stripped-down, weight-free evaluation. They fall into four performance groups. The grouping tracks how much each measure relies on verbs that occur with both nouns. Building on this, and on the observation that substitutability is often one-directional, Lee introduces an asymmetric smoothed KL variant, the skew divergence. It outperforms all the others and cuts error by about 4% on average relative to the Jensen-Shannon divergence. The work offers both a better metric and a principled way to classify similarity functions.

## Key points
- **Data:** verb–object cooccurrences drawn from the 1988 AP newswire, limited to pairs whose noun is among the 1000 most frequent, and obtained using Church's and Yarowsky's tools.
  - 587,833 pairs (80%) were used for training.
  - Test sets were built from the remaining 20% after removing pairs already seen in training, then split into five partitions.
- **Measures compared:** Euclidean (L2) distance, L1 norm, cosine, Jaccard's coefficient, Jensen-Shannon divergence, confusion probability, and Kendall's τ.
  - Every one of them stays far below Katz backoff's 51% error rate.
- **Performance groups:**
  1. L2 (worst)
  2. Kendall's τ
  3. confusion probability and cosine
  4. L1, Jensen-Shannon, and Jaccard (best)
- **Skew divergence:** defined as the KL divergence of r from a mixture αq + (1−α)r. With α = 0.99, it beats Jensen-Shannon by about 4% average error reduction (significant at .01). It also beats Jaccard significantly under a paired t-test at nearly all k.
- **Measures excluded and why:**
  - Mutual information compares a joint distribution with a product distribution, not two arbitrary distributions.
  - Dice is monotonic in Jaccard, so it would be redundant.
  - Hamming-type metrics are meant for symbolic features, and the value difference metric needs probability-labelled training data.
  - Plain KL needs a smoothed base model.

## Interesting ideas
- **Separate ranking quality from weighting.** Instead of judging similarity functions inside the full smoothing model, count how many of the top-k neighbours make the correct decision. This shows whether a metric ranks well, independent of the weighting scheme. The same trick applies to evaluating any retrieval or neighbour-selection component.
- **Decompose measures by support regions.** Rewriting each function as sums over shared versus non-shared contexts explains why some measures behave alike. This analytic lens could help compare embedding or feature similarity metrics.
- **Rely mainly on shared contexts.** The best functions depend mostly on contexts both words share, which is also computationally cheap because that set is small. This argues for discounting purely negative evidence in sparse settings.
- **Use asymmetric similarity for proxies.** A broad word like "fruit" can stand in for "apple" more easily than the reverse. Directional measures may therefore suit tasks such as hypernymy or backoff to more general categories.
- **Smooth one distribution with the other.** Mixing a small portion of the second distribution into the first keeps KL defined on sparse counts without a separately smoothed base model. This is a cheap fix wherever zero probabilities break divergence computations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Institutions

- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 68 works · 52,012 citations

## Cite

Cite as: Lillian Lee. 1999. Measures of Distributional Similarity. Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lee1999measures,
  title = {{Measures of Distributional Similarity}},
  author = {Lillian Lee},
  booktitle = {Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics},
  year = {1999},
  doi = {10.3115/1034678.1034693},
  url = {https://doi.org/10.3115/1034678.1034693}
}
```

- [← previous Automatic Identification of Word Translations from Unrelated English and German Corpora](https://md.research.elamur.ai/free/en/papers/rapp-1999-automatic-identification-word-translations-unrelated/)
- [next → Representing Text Chunks](https://md.research.elamur.ai/free/en/papers/sang-1999-representing-text-chunks/)
- [↑ back to papers of 1999](https://md.research.elamur.ai/free/en/papers/1999/)

## References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Claire Cardie. 1993. A case-based approach to knowledge acquisition for domain-specific sentence analysis. In 11th National Conference on Artifical Intelligence, pages 798-803. [find](https://scholar.google.com/scholar?q=Claire+Cardie%2E+1993%2E+A+case%2Dbased+approach+to+knowledge+acquisition+for+domain%2Dspecific+sentence+analysis%2E+In+11th+National+Conference+on+Artifical+Intelligence%2C+pages+798%2D803%2E)
2. Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29. [our card](https://md.research.elamur.ai/free/en/papers/church-1990-word-association-norms-mutual-information/)
3. Kenneth W. Church. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In Second Conference on Applied Natural Language Processing, pages 136-143. [our card](https://md.research.elamur.ai/free/en/papers/church-1988-stochastic-parts-program-noun-phrase/)
4. Thomas M. Cover and Joy A. Thomas. 1991. Elements of Information Theory. John Wiley. [find](https://scholar.google.com/scholar?q=Thomas+M%2E+Cover+and+Joy+A%2E+Thomas%2E+1991%2E+Elements+of+Information+Theory%2E+John+Wiley%2E)
5. Ido Dagan, Shanl Marcus, and Shanl Markovitch. 1995. Contextual word similarity and estimation from sparse data. Computer Speech and Language, 9:123-152. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Shanl+Marcus%2C+and+Shanl+Markovitch%2E+1995%2E+Contextual+word+similarity+and+estimation+from+sparse+data%2E+Computer+Speech+and+Language%2C+9%3A123%2D152%2E)
6. Ido Dagan, Lillian Lee, and Fernando Pereira. 1999. Similarity-based models of cooccurrence probabilities. Machine Learning, 34(1-3) :43-69. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Lillian+Lee%2C+and+Fernando+Pereira%2E+1999%2E+Similarity%2Dbased+models+of+cooccurrence+probabilities%2E+Machine+Learning%2C+34%281%2D3%29+%3A43%2D69%2E)
7. Ute Essen and Volker Steinbiss. 1992. Co-occurrence smoothing for stochastic language modeling. In ICASSP 92, volume 1, pages 161-164. [find](https://scholar.google.com/scholar?q=Ute+Essen+and+Volker+Steinbiss%2E+1992%2E+Co%2Doccurrence+smoothing+for+stochastic+language+modeling%2E+In+ICASSP+92%2C+volume+1%2C+pages+161%2D164%2E)
8. Jean Dickinson Gibbons. 1993. Nonparametric Measures of Association. Sage University Paper series on Quantitative Applications in the Social Sciences, 07-091. Sage Publications. [find](https://scholar.google.com/scholar?q=Jean+Dickinson+Gibbons%2E+1993%2E+Nonparametric+Measures+of+Association%2E+Sage+University+Paper+series+on+Quantitative+Applications+in+the+Social+Sciences%2C+07%2D091%2E+Sage+Publications%2E)
9. Ralph Grishman and John Sterling. 1993. Smoothing of automatically generated selectional constraints. In Human Language Technology: Proceedings of the ARPA Workshop, pages 254-259. [find](https://scholar.google.com/scholar?q=Ralph+Grishman+and+John+Sterling%2E+1993%2E+Smoothing+of+automatically+generated+selectional+constraints%2E+In+Human+Language+Technology%3A+Proceedings+of+the+ARPA+Workshop%2C+pages+254%2D259%2E)
10. Vasileios Hatzivassiloglou and Kathleen McKeown. 1993. Towards the automatic identification of adjectival scales: Clustering of adjectives according to meaning. In 31st Annual Meeting of the ACL, pages 172-182. [find](https://scholar.google.com/scholar?q=Vasileios+Hatzivassiloglou+and+Kathleen+McKeown%2E+1993%2E+Towards+the+automatic+identification+of+adjectival+scales%3A+Clustering+of+adjectives+according+to+meaning%2E+In+31st+Annual+Meeting+of+the+ACL%2C+page)
11. Vasileios Hatzivassiloglou. 1996. Do we need linguistics when we have statistics? A comparative analysis of the contributions of linguistic cues to a statistical word grouping system. In Judith L. Klavans and Philip Resnik, editors, The Balancing Act, pages 67-94. MIT Press. [find](https://scholar.google.com/scholar?q=Vasileios+Hatzivassiloglou%2E+1996%2E+Do+we+need+linguistics+when+we+have+statistics%3F+A+comparative+analysis+of+the+contributions+of+linguistic+cues+to+a+statistical+word+grouping+system%2E+In+Judith+L%2E+Kla)
12. Don Hindle. 1990. Noun classification from predicate-argument structures. In 28th Annual Meeting of the A CL, pages 268-275. [our card](https://md.research.elamur.ai/free/en/papers/hindle-1990-noun-classification-predicate-argument-structures/)
13. Frederick Jelinek and Robert L. Mercer. 1980. Interpolated estimation of Markov source parameters from sparse data. In Proceedings of the Workshop on Pattern Recognition in Practice. [find](https://scholar.google.com/scholar?q=Frederick+Jelinek+and+Robert+L%2E+Mercer%2E+1980%2E+Interpolated+estimation+of+Markov+source+parameters+from+sparse+data%2E+In+Proceedings+of+the+Workshop+on+Pattern+Recognition+in+Practice%2E)
14. William P. Jones and George W. Furnas. 1987. Pictures of relevance. Journal of the American Society for Information Science, 38(6):420-442. [find](https://scholar.google.com/scholar?q=William+P%2E+Jones+and+George+W%2E+Furnas%2E+1987%2E+Pictures+of+relevance%2E+Journal+of+the+American+Society+for+Information+Science%2C+38%286%29%3A420%2D442%2E)
15. Yael Karov and Shimon Edelman. 1998. Similarity-based word sense disambiguation. Computational Linguistics, 24(1):41-59. [find](https://scholar.google.com/scholar?q=Yael+Karov+and+Shimon+Edelman%2E+1998%2E+Similarity%2Dbased+word+sense+disambiguation%2E+Computational+Linguistics%2C+24%281%29%3A41%2D59%2E)
16. Slava M. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-35(3):400--401, March. [find](https://scholar.google.com/scholar?q=Slava+M%2E+Katz%2E+1987%2E+Estimation+of+probabilities+from+sparse+data+for+the+language+model+component+of+a+speech+recognizer%2E+IEEE+Transactions+on+Acoustics%2C+Speech+and+Signal+Processing%2C+ASSP%2D35%283%29%3A400%2D)
17. Leonard Kanfman and Peter J. Rousseeuw. 1990. Finding Groups in Data: An Introduction to Cluster Analysis. John Wiley and Sons. [find](https://scholar.google.com/scholar?q=Leonard+Kanfman+and+Peter+J%2E+Rousseeuw%2E+1990%2E+Finding+Groups+in+Data%3A+An+Introduction+to+Cluster+Analysis%2E+John+Wiley+and+Sons%2E)
18. Lillian Lee. 1997. Similarity-Based Approaches to Natural Language Processing. Ph.D. thesis, Harvard University. [find](https://scholar.google.com/scholar?q=Lillian+Lee%2E+1997%2E+Similarity%2DBased+Approaches+to+Natural+Language+Processing%2E+Ph%2ED%2E+thesis%2C+Harvard+University%2E)
19. Dekang Lin. 1997. Using syntactic dependency as local context to resolve word sense ambiguity. In 35th Annual Meeting of the ACL, pages 64-71. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1997%2E+Using+syntactic+dependency+as+local+context+to+resolve+word+sense+ambiguity%2E+In+35th+Annual+Meeting+of+the+ACL%2C+pages+64%2D71%2E)
20. Dekang Lin. 1998a. Automatic retrieval and clustering of similar words. In COLING-A CL '98, pages 768-773. [our card](https://md.research.elamur.ai/free/en/papers/lin-1998-automatic-retrieval-clustering-similar-words/)
21. Dekang Lin. 1998b. An information theoretic definition of similarity. In Machine Learning: Proceedings of the Fiftheenth International Conference (ICML '98). [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1998b%2E+An+information+theoretic+definition+of+similarity%2E+In+Machine+Learning%3A+Proceedings+of+the+Fiftheenth+International+Conference+%28ICML+%2798%29%2E)
22. Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1):145-151. [find](https://scholar.google.com/scholar?q=Jianhua+Lin%2E+1991%2E+Divergence+measures+based+on+the+Shannon+entropy%2E+IEEE+Transactions+on+Information+Theory%2C+37%281%29%3A145%2D151%2E)
23. Alpha K. Luk. 1995. Statistical sense disambiguation with relatively small corpora using dictionary definitions. In 33rd Annual Meeting of the ACL, pages 181-188. [find](https://scholar.google.com/scholar?q=Alpha+K%2E+Luk%2E+1995%2E+Statistical+sense+disambiguation+with+relatively+small+corpora+using+dictionary+definitions%2E+In+33rd+Annual+Meeting+of+the+ACL%2C+pages+181%2D188%2E)
24. Hwee Tou Ng and Hian Beng Lee. 1996. Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach. In 3~th Annual Meeting of the ACL, pages 40--47. [our card](https://md.research.elamur.ai/free/en/papers/ng-1996-integrating-multiple-knowledge-sources-disambiguate/)
25. Hwee Tou Ng. 1997. Exemplar-based word sense disambiguation: Some recent improvements. In Second Conference on Empirical Methods in Natural Language Processing (EMNLP-2), pages 208-213. [find](https://scholar.google.com/scholar?q=Hwee+Tou+Ng%2E+1997%2E+Exemplar%2Dbased+word+sense+disambiguation%3A+Some+recent+improvements%2E+In+Second+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%2D2%29%2C+pages+208%2D213%2E)
26. C. Radhakrishna Rao. 1982. Diversity: Its measurement, decomposition, apportionment and analysis. SankyhZt: The Indian Journal of Statistics, 44(A):1-22. [find](https://scholar.google.com/scholar?q=C%2E+Radhakrishna+Rao%2E+1982%2E+Diversity%3A+Its+measurement%2C+decomposition%2C+apportionment+and+analysis%2E+SankyhZt%3A+The+Indian+Journal+of+Statistics%2C+44%28A%29%3A1%2D22%2E)
27. Philip Resnik. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of IJCAI-95, pages 448-453. [our card](https://md.research.elamur.ai/free/en/papers/resnik-1995-information-content-evaluate-semantic-similarity/)
28. Gerard Salton and Michael J. McGill. 1983. Introduction to Modern Information Retrieval. [find](https://scholar.google.com/scholar?q=Gerard+Salton+and+Michael+J%2E+McGill%2E+1983%2E+Introduction+to+Modern+Information+Retrieval%2E)
29. Frank Smadja, Kathleen R. McKeown, and Vasileios Hatzivassiloglou. 1996. Translating collocations for bilingual lexicons: A statistical approach. Computational Linguistics, 22(1):1-38. [our card](https://md.research.elamur.ai/free/en/papers/smadja-1996-translating-collocations-bilingual-lexicons-statistical/)
30. Craig Stanfill and David Waltz. 1986. Toward memory-based reasoning. Communications of the ACM, 29(12):1213-1228. [find](https://scholar.google.com/scholar?q=Craig+Stanfill+and+David+Waltz%2E+1986%2E+Toward+memory%2Dbased+reasoning%2E+Communications+of+the+ACM%2C+29%2812%29%3A1213%2D1228%2E)
31. K. Sugawara, M. Nishimura, K. Toshioka, M. Okochi, and T. Kaneko. 1985. Isolated word recognition using hidden Markov models. In ICASSP 85, pages 1-4. [find](https://scholar.google.com/scholar?q=K%2E+Sugawara%2C+M%2E+Nishimura%2C+K%2E+Toshioka%2C+M%2E+Okochi%2C+and+T%2E+Kaneko%2E+1985%2E+Isolated+word+recognition+using+hidden+Markov+models%2E+In+ICASSP+85%2C+pages+1%2D4%2E)
32. C. J. van Rijsbergen. 1979. Information Retrieval. Butterworths, second edition. [find](https://scholar.google.com/scholar?q=C%2E+J%2E+van+Rijsbergen%2E+1979%2E+Information+Retrieval%2E+Butterworths%2C+second+edition%2E)
33. Jakub Zavrel and Walter Daelemans. 1997. Memory-based learning: Using similarity for smoothing. In 35th Annual Meeting of the A CL, pages 436-443. [find](https://scholar.google.com/scholar?q=Jakub+Zavrel+and+Walter+Daelemans%2E+1997%2E+Memory%2Dbased+learning%3A+Using+similarity+for+smoothing%2E+In+35th+Annual+Meeting+of+the+A+CL%2C+pages+436%2D443%2E)