---
title: "A Perspective on Word Sense Disambiguation Methods and Their Evaluation — Resnik 1997"
description: "They then propose remedies: Score systems with a cross-entropy-style measure so that well-calibrated probabilities… 28 references, cited by 39 papers here."
authors: ["Philip Resnik", "David Yarowsky"]
year: 1997
venue: "Tagging Text with Lexical Semantics: Why, What, and How?"
source: "https://aclanthology.org/W97-0213/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/resnik-1997-perspective-word-sense-disambiguation-methods/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/resnik-1997-perspective-word-sense-disambiguation-methods/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1997](https://md.research.elamur.ai/free/en/papers/1997/) / A Perspective on Word Sense Disambiguation Methods and Their Evaluation

# A Perspective on Word Sense Disambiguation Methods and Their Evaluation

**Authors:** [Philip Resnik](https://md.research.elamur.ai/free/en/authors/philip-resnik/) and [David Yarowsky](https://md.research.elamur.ai/free/en/authors/david-yarowsky/) · **Year:** 1997 · **Venue:** Tagging Text with Lexical Semantics: Why, What, and How? · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W97-0213/) · [PDF](https://aclanthology.org/W97-0213.pdf) · [OpenAlex](https://openalex.org/W91190427)

**Citations:** 112 citations (OpenAlex, 2026-10-14) · [39](https://md.research.elamur.ai/free/en/papers/resnik-1997-perspective-word-sense-disambiguation-methods/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
This position paper argues that word sense disambiguation (WSD) remains a largely unsolved lexical problem, unlike POS tagging or morphology. One major reason is that the field lacks shared evaluation practice: studies use different target words, corpora and metrics. The authors make four observations:
- evaluation is unstandardized;
- the value of WSD differs by application, with machine translation looking more promising than IR or speech recognition;
- sense-annotated data are scarce;
- methods have barely converged, since WSD draws on many knowledge sources and is highly word-specific.

They then propose remedies:
- Score systems with a cross-entropy-style measure so that well-calibrated probabilities earn partial credit, rather than relying only on exact-match accuracy.
- Weight errors by semantic distance between senses in hierarchical inventories.
- Build a common evaluation setup that combines the traditional focus on selected words with broad coverage.
- Limit sense inventories to distinctions that languages commonly express with different words. Aligned parallel corpora could then supply tagged data without manual annotation.

The paper matters because it shaped the evaluation agenda later adopted in shared WSD tasks.

## Key points
- Earlier shared test data were rare. One exception is the 2094-instance *line* dataset of Leacock et al. (1993), reused by Lehman (1994) and Mooney (1996). The WordNet semantic concordance (Miller et al., 1994) is the only all-words annotation, but it typically yields fewer than 10 examples per polysemous word.
- In the *interest* example with four hypothetical systems, all of them choose the wrong sense. Exact match scores them identically. Under cross-entropy, System 1, which gives probability .42 to the correct sense, costs 1.25 bits. A system that gives the correct sense zero probability receives an infinite penalty.
- For systems without reliable probabilities, the authors suggest a variant without the log term. It reduces to simple % correct when outputs are 0/1.
- A hierarchical inventory for *bank* (financial vs. geographical, with subsenses) illustrates distance-sensitive penalties.
- Translation distinctions can serve as sense labels, for example *duty* → French *devoir*/*droit*, German *Tisch* vs. *Tischrunde*, and Spanish *dedo* (finger/toe). The authors propose mapping WordNet, LDOCE and COBUILD senses to such distinctions, for instance via EuroWordNet (Spanish, Italian, Dutch, English).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🔀 Word sense disambiguation](https://md.research.elamur.ai/free/en/topics/word-sense-disambiguation/) · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

## Cite

Cite as: Philip Resnik and David Yarowsky. 1997. A Perspective on Word Sense Disambiguation Methods and Their Evaluation. Tagging Text with Lexical Semantics: Why, What, and How?.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{resnik1997perspective,
  title = {{A Perspective on Word Sense Disambiguation Methods and Their Evaluation}},
  author = {Philip Resnik and David Yarowsky},
  booktitle = {Tagging Text with Lexical Semantics: Why, What, and How?},
  year = {1997},
  url = {https://aclanthology.org/W97-0213/}
}
```

- [← previous IGTree: Using Trees for Compression and Classification in Lazy Learning Algorithms](https://md.research.elamur.ai/free/en/papers/daelemans-1997-igtree-trees-compression-classification-lazy/)
- [next → Event coreference for information extraction](https://md.research.elamur.ai/free/en/papers/humphreys-1997-event-coreference-information-extraction/)
- [↑ back to papers of 1997](https://md.research.elamur.ai/free/en/papers/1997/)

## References

28 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. L. Bahl and R. Mercer. 1976. Part-of-speech as-sigument by a statistical decision algorithm. In International Symposium on Information Theory, Ronneby, Sweden. [find](https://scholar.google.com/scholar?q=L%2E+Bahl+and+R%2E+Mercer%2E+1976%2E+Part%2Dof%2Dspeech+as%2Dsigument+by+a+statistical+decision+algorithm%2E+In+International+Symposium+on+Information+Theory%2C+Ronneby%2C+Sweden%2E)
2. L. Bald, F. Jelinek, and R. Mercer. 1983. A maxi m u m likelihood approach to continuous speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-5(2):179-190. [find](https://scholar.google.com/scholar?q=L%2E+Bald%2C+F%2E+Jelinek%2C+and+R%2E+Mercer%2E+1983%2E+A+maxi+m+u+m+likelihood+approach+to+continuous+speech+recognition%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+PAMI%2D5%282%29%3A179%2D190%2E)
3. L. Bloksma, P. Dfez-Orzas and P. Vossen. 1996. User Requirements and Functional Specification of the EuroWordNet Project. http:/lw~m, let. uva. nll-er, m. [find](https://scholar.google.com/scholar?q=L%2E+Bloksma%2C+P%2E+Dfez%2DOrzas+and+P%2E+Vossen%2E+1996%2E+User+Requirements+and+Functional+Specification+of+the+EuroWordNet+Project%2E+http%3A%2Flw%7Em%2C+let%2E+uva%2E+nll%2Der%2C+m%2E)
4. E. Brill. 1993. A Corpus-Based Approach to Language Learning. Ph.D. thesis, Computer and Information Science, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=E%2E+Brill%2E+1993%2E+A+Corpus%2DBased+Approach+to+Language+Learning%2E+Ph%2ED%2E+thesis%2C+Computer+and+Information+Science%2C+University+of+Pennsylvania%2E)
5. P. Brown, V. Della Pietra, P. deSouza, J. Lai, and R. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467-480. [find](https://scholar.google.com/scholar?q=P%2E+Brown%2C+V%2E+Della+Pietra%2C+P%2E+deSouza%2C+J%2E+Lai%2C+and+R%2E+Mercer%2E+1992%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+Linguistics%2C+18%284%29%3A467%2D480%2E)
6. R. Bruce and J. Wiebe. 1994. Word-sense disambiguation using decomposable models. In Proceedings of the 3~nd Annual Meeting of the Association .for Computational Linguistics, Las Cruces. [find](https://scholar.google.com/scholar?q=R%2E+Bruce+and+J%2E+Wiebe%2E+1994%2E+Word%2Dsense+disambiguation+using+decomposable+models%2E+In+Proceedings+of+the+3%7End+Annual+Meeting+of+the+Association+%2Efor+Computational+Linguistics%2C+Las+Cruces%2E)
7. K. Church. 1988. A stochastic parts program and noun phrase parser for unrestricted texts. In Proceedings of the Second Conference on Applied Natural Language Processing, Austin, Texas. [find](https://scholar.google.com/scholar?q=K%2E+Church%2E+1988%2E+A+stochastic+parts+program+and+noun+phrase+parser+for+unrestricted+texts%2E+In+Proceedings+of+the+Second+Conference+on+Applied+Natural+Language+Processing%2C+Austin%2C+Texas%2E)
8. B. Dorr and D. Jones. 1996a. Acquisition of semantic lexicons: Using word sense disarnbiguation to improve precision. In Proceedings of the SIGLEX Workshop on Breadth and Depth of Semantic Lexicons, Santa Cruz, CA. [find](https://scholar.google.com/scholar?q=B%2E+Dorr+and+D%2E+Jones%2E+1996a%2E+Acquisition+of+semantic+lexicons%3A+Using+word+sense+disarnbiguation+to+improve+precision%2E+In+Proceedings+of+the+SIGLEX+Workshop+on+Breadth+and+Depth+of+Semantic+Lexicons%2C+S)
9. B. Dorr and D. Jones. 1996b. Role of word sense disambiguation in lexical acquisition: Predicting semantics from syntactic cues. In Proceedings of the International Conference on Computational Linguistics, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=B%2E+Dorr+and+D%2E+Jones%2E+1996b%2E+Role+of+word+sense+disambiguation+in+lexical+acquisition%3A+Predicting+semantics+from+syntactic+cues%2E+In+Proceedings+of+the+International+Conference+on+Computational+Linguis)
10. W. Gale, K. Church, and D. Yarowsky. 1992. One sense per discourse. Proceedings of the 4th DARPA Speech and Natural Language Workshop. [find](https://scholar.google.com/scholar?q=W%2E+Gale%2C+K%2E+Church%2C+and+D%2E+Yarowsky%2E+1992%2E+One+sense+per+discourse%2E+Proceedings+of+the+4th+DARPA+Speech+and+Natural+Language+Workshop%2E)
11. W. Gale, K. Church, and D. Yarowsky. A method for disambiguating word senses in a large corpus. Computers and the Humanities, 26:415-439, 1992. [find](https://scholar.google.com/scholar?q=W%2E+Gale%2C+K%2E+Church%2C+and+D%2E+Yarowsky%2E+A+method+for+disambiguating+word+senses+in+a+large+corpus%2E+Computers+and+the+Humanities%2C+26%3A415%2D439%2C+1992%2E)
12. F. Jelinek. 1985. Markov source modeling of text generation. In J. Skwirzinski, editor, Impact of Processing Techniques on Communication. Dordrecht. [find](https://scholar.google.com/scholar?q=F%2E+Jelinek%2E+1985%2E+Markov+source+modeling+of+text+generation%2E+In+J%2E+Skwirzinski%2C+editor%2C+Impact+of+Processing+Techniques+on+Communication%2E+Dordrecht%2E)
13. S. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-35(3):400-401. [find](https://scholar.google.com/scholar?q=S%2E+Katz%2E+1987%2E+Estimation+of+probabilities+from+sparse+data+for+the+language+model+component+of+a+speech+recognizer%2E+IEEE+Transactions+on+Acoustics%2C+Speech+and+Signal+Processing%2C+ASSP%2D35%283%29%3A400%2D401%2E)
14. It. Krovetz and W. B. Croft. 1992. Lexical ambiguity and information retrieval. ACM Transactions on Information Systems, 10(2):115-141. [find](https://scholar.google.com/scholar?q=It%2E+Krovetz+and+W%2E+B%2E+Croft%2E+1992%2E+Lexical+ambiguity+and+information+retrieval%2E+ACM+Transactions+on+Information+Systems%2C+10%282%29%3A115%2D141%2E)
15. C. Leaeock, G. Towell and E. Voorhees. 1993. Corpus-based statistical sense resolution. In Proceedings, ARPA Human Language Technology Workshop, pp. 260-265, Plainsboro, NJ. [find](https://scholar.google.com/scholar?q=C%2E+Leaeock%2C+G%2E+Towell+and+E%2E+Voorhees%2E+1993%2E+Corpus%2Dbased+statistical+sense+resolution%2E+In+Proceedings%2C+ARPA+Human+Language+Technology+Workshop%2C+pp%2E+260%2D265%2C+Plainsboro%2C+NJ%2E)
16. J. Lehman. 1994. Toward the essential nature of statistical knowledge in sense resolution. In Proceedings of the TvJelfth National Conference on Artificial Intelligence, pp. 734-471. [find](https://scholar.google.com/scholar?q=J%2E+Lehman%2E+1994%2E+Toward+the+essential+nature+of+statistical+knowledge+in+sense+resolution%2E+In+Proceedings+of+the+TvJelfth+National+Conference+on+Artificial+Intelligence%2C+pp%2E+734%2D471%2E)
17. R. Mooney. 1996. Comparative experiments on disam~i~mating word senses: An illustration of the role of bias in machine learning. In Proceedings of the Con]erence on Empirical Methods in Natural Language Processing, Philadelphia. [find](https://scholar.google.com/scholar?q=R%2E+Mooney%2E+1996%2E+Comparative+experiments+on+disam%7Ei%7Emating+word+senses%3A+An+illustration+of+the+role+of+bias+in+machine+learning%2E+In+Proceedings+of+the+Con%5Derence+on+Empirical+Methods+in+Natural+Langua)
18. S. McRoy. 1992. Using multiple knowledge sources for word sense discrimination. Computational Linguistics, 18(1):1-30. [find](https://scholar.google.com/scholar?q=S%2E+McRoy%2E+1992%2E+Using+multiple+knowledge+sources+for+word+sense+discrimination%2E+Computational+Linguistics%2C+18%281%29%3A1%2D30%2E)
19. G. Miller and W. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1-28. [find](https://scholar.google.com/scholar?q=G%2E+Miller+and+W%2E+Charles%2E+1991%2E+Contextual+correlates+of+semantic+similarity%2E+Language+and+Cognitive+Processes%2C+6%281%29%3A1%2D28%2E)
20. G. Miller, M. Chodorow, S. Landes, C. Leacock, and R. Thomas. 1994. Using a semantic concordance for sense identification. In Proceedings of the ARPA Human Language Technology Workshop, San Francisco. Morgan Kaufmarm. [our card](https://md.research.elamur.ai/free/en/papers/miller-1994-semantic-concordance-sense-identification/)
21. H. Ng and H. Lee. 1996. Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach. In Proceedings o.f the 34th Annual Meeting o.f the Society for Computational Linguistics, pp. 40-47, Santa Cruz, CA. 1993. Selection and ln]ormation: A Class-Based Approach to Le~ca… [our card](https://md.research.elamur.ai/free/en/papers/ng-1996-integrating-multiple-knowledge-sources-disambiguate/)
22. P. Resnik. Ph.D. thesis, University of Pennsylvania. ( f t p : / / ftp. cis. upenn, edu/pub/ircs/tr/93-42, ps. Z). [find](https://scholar.google.com/scholar?q=P%2E+Resnik%2E+Ph%2ED%2E+thesis%2C+University+of+Pennsylvania%2E+%28+f+t+p+%3A+%2F+%2F+ftp%2E+cis%2E+upenn%2C+edu%2Fpub%2Fircs%2Ftr%2F93%2D42%2C+ps%2E+Z%29%2E)
23. P. Resnik. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of the I~th International Joint Conference on Artificial Intelligence (IJCAI-95). (cmp-lg/9511007). [our card](https://md.research.elamur.ai/free/en/papers/resnik-1995-information-content-evaluate-semantic-similarity/)
24. R. Richardson, A. Smeaton, and J. Murphy. 1994. Using WordNet as a knowledge base for measuring semantic similarity between words. Working Paper CA-1294, Dublin City University, School of Computer Applications, Dublin, Ireland. f t p : / / f t p , compapp, dcu. ie/pub/w-papers/ 1994/CA1294. ps. Z. [find](https://scholar.google.com/scholar?q=R%2E+Richardson%2C+A%2E+Smeaton%2C+and+J%2E+Murphy%2E+1994%2E+Using+WordNet+as+a+knowledge+base+for+measuring+semantic+similarity+between+words%2E+Working+Paper+CA%2D1294%2C+Dublin+City+University%2C+School+of+Computer+App)
25. P. Tapanainen and A. Voutilainen. 1994. Tagging accurately - don't guess if you know. In Proceedings of ANLP '94. [find](https://scholar.google.com/scholar?q=P%2E+Tapanainen+and+A%2E+Voutilainen%2E+1994%2E+Tagging+accurately+%2D+don%27t+guess+if+you+know%2E+In+Proceedings+of+ANLP+%2794%2E)
26. Y. Wilks and M. Stevenson. 1996. The grammar of sense: Is word-sense tagging much more than part-of-speech tagging? cmp-lg/9607028. [find](https://scholar.google.com/scholar?q=Y%2E+Wilks+and+M%2E+Stevenson%2E+1996%2E+The+grammar+of+sense%3A+Is+word%2Dsense+tagging+much+more+than+part%2Dof%2Dspeech+tagging%3F+cmp%2Dlg%2F9607028%2E)
27. D. Yarowsky. 1992. Word-sense disambiguation using statistical models of Roget's categories trained on large corpora. In Proceedings of COLING-9~, pp. 454-460, Nantes, Prance. [our card](https://md.research.elamur.ai/free/en/papers/yarowsky-1992-word-sense-disambiguation-statistical-models/)
28. D. Yarowsky. 1993. One sense per collocation. Proccedings of the ARIA Human Language Technology Workshop, Morgan Kaufmann, pp. 266-271. [find](https://scholar.google.com/scholar?q=D%2E+Yarowsky%2E+1993%2E+One+sense+per+collocation%2E+Proccedings+of+the+ARIA+Human+Language+Technology+Workshop%2C+Morgan+Kaufmann%2C+pp%2E+266%2D271%2E)