---
title: "Measuring the Semantic Similarity of Texts — Corley 2005"
description: "Corley and Mihalcea propose a knowledge-based way to lift word-level semantic similarity to whole… 68.8% accuracy. 21 references, cited by 63 papers here."
authors: ["Courtney Corley", "Rada Mihalcea"]
year: 2005
venue: "Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment"
source: "https://aclanthology.org/W05-1203/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/corley-2005-measuring-semantic-similarity-texts/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/corley-2005-measuring-semantic-similarity-texts/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2005](https://md.research.elamur.ai/free/en/papers/2005/) / Measuring the Semantic Similarity of Texts

# Measuring the Semantic Similarity of Texts

**Authors:** Courtney Corley and [Rada Mihalcea](https://md.research.elamur.ai/free/en/authors/rada-mihalcea/) · **Year:** 2005 · **Venue:** Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W05-1203/) · [PDF](https://aclanthology.org/W05-1203.pdf) · [OpenAlex](https://openalex.org/W1990524510)

**Citations:** 352 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/corley-2005-measuring-semantic-similarity-texts/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Classic text similarity counts shared words, so two sentences that express the same idea in different vocabulary look unrelated. Corley and Mihalcea propose a knowledge-based way to lift word-level semantic similarity to whole text segments. Each text is split into part-of-speech sets. For each noun and verb, the most similar word in the other text is found with one of six WordNet-based measures from the WordNet::Similarity package: Jiang & Conrath, Lin, Resnik, Wu & Palmer, Lesk, or Leacock & Chodorow. Adjectives, adverbs and numbers are matched only by surface form. Each match is weighted by inverse document frequency from the British National Corpus and normalized, which gives a directional score; averaging both directions gives a symmetric score. The method was tested on paraphrase and entailment recognition. It beats random, lexical-overlap and tf.idf cosine baselines, and averaging all six metrics reaches 68.8% (unsupervised) and 71.5% (supervised) accuracy on paraphrases. The work shows that cheap, interpretable lexical resources can capture meaning beyond surface overlap.

## Key points
- **Six WordNet measures compared.** Each of the six measures (Jiang & Conrath, Resnik, Lin, Lesk, Wu & Palmer and Leacock & Chodorow) is converted from concept-level to word-level similarity by picking the sense pair with the highest score. Only nouns and verbs get semantic matching, because most of these measures do not cover adjectives and adverbs.
- **Paraphrase results.** In the unsupervised setting, a pair counts as positive if its score exceeds 0.5.
  - The combined metric reaches 68.8% accuracy.
  - The lexical-matching baseline gets 66.1% and the tf.idf cosine baseline 65.4%.
  - In the supervised setting, the best result is 71.5%, combining the metrics with lexical matching.
- **Entailment results.** Only the direction toward the hypothesis is scored.
  - Accuracy is 58.3% unsupervised and 58.9% supervised.
  - The authors call these figures competitive with the best PASCAL entailment systems.
- **Statistical significance.** The gain of the combined metric over lexical matching was significant in all experiments (paired t-test, p \< 0.001).
- **LSA comparison.** LSA was also tried as a corpus-based alternative, but it scored below the lexical baseline, so its numbers are not reported.

## Interesting ideas
- **Directional scoring.** Computing similarity of one text relative to the other, then averaging both directions, gives one framework for symmetric tasks (paraphrase) and asymmetric ones (entailment, where only the hypothesis side matters). The same design could serve query–document matching or checking whether a summary is covered by its source.
- **Specificity weighting.** Weighting each word match by its idf means that aligning two rare, specific words counts much more than aligning vague, frequent verbs. This cheap trick can be added to any alignment-based similarity, including ones built on embeddings.
- **Type-dependent matching.** Semantic matching is used only where the resource is reliable (nouns, verbs), and exact matching is used elsewhere. Numbers get their own set because they often decide meaning. Choosing the matching strategy by word type is a practical pattern for languages or domains where lexical resources cover some categories better than others.
- **Averaging an ensemble of measures.** A plain average of several imperfect word-similarity metrics beat each individual metric without any training. This makes it a robust default when labelled data for tuning is unavailable.
- **Interpretable alignments.** Every text score breaks down into explicit word pairs with their individual scores, as in the paper's walk-through table. Unlike black-box methods such as LSA, this makes errors easy to inspect and debug.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Institutions

- [University of North Texas](https://md.research.elamur.ai/free/en/universities/university-of-north-texas/) — University, Denton, United States · 13 works · 3,921 citations

## Cite

Cite as: Courtney Corley and Rada Mihalcea. 2005. Measuring the Semantic Similarity of Texts. Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{corley2005measuring,
  title = {{Measuring the Semantic Similarity of Texts}},
  author = {Courtney Corley and Rada Mihalcea},
  booktitle = {Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment},
  year = {2005},
  doi = {10.3115/1631862.1631865},
  url = {https://doi.org/10.3115/1631862.1631865}
}
```

- [← previous A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005](https://md.research.elamur.ai/free/en/papers/tseng-2005-conditional-random-field-word-segmenter/)
- [next → Representing Discourse Coherence: A Corpus-Based Study](https://md.research.elamur.ai/free/en/papers/wolf-2005-representing-discourse-coherence-corpus-based/)
- [↑ back to papers of 2005](https://md.research.elamur.ai/free/en/papers/2005/)

## References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Budanitsky and G. Hirst. 2001. Semantic distance in wordnet: An experimental, application-oriented evaluation of five measures. In Proceedings of the NAACL Workshop on Word-Net and Other Lexical Resources, Pittsburgh, June. [find](https://scholar.google.com/scholar?q=A%2E+Budanitsky+and+G%2E+Hirst%2E+2001%2E+Semantic+distance+in+wordnet%3A+An+experimental%2C+application%2Doriented+evaluation+of+five+measures%2E+In+Proceedings+of+the+NAACL+Workshop+on+Word%2DNet+and+Other+Lexical+Re)
2. I. Dagan, O. Glickman, and B. Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of the PASCAL Workshop. [find](https://scholar.google.com/scholar?q=I%2E+Dagan%2C+O%2E+Glickman%2C+and+B%2E+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+PASCAL+Workshop%2E)
3. W.B. Dolan, C. Quirk, and C. Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th International Conference on Computational Linguistics, Geneva, Switzerland. [our card](https://md.research.elamur.ai/free/en/papers/dolan-2004-unsupervised-construction-large-paraphrase-corpora/)
4. Y. Freund and R.E. Schapire. 1998. Large margin classification using the perceptron algorithm. In Proceedings of the 11th Annual Conference on Computational Learning Theory, pages 209–217, New York, NY. ACM Press. [find](https://scholar.google.com/scholar?q=Y%2E+Freund+and+R%2EE%2E+Schapire%2E+1998%2E+Large+margin+classification+using+the+perceptron+algorithm%2E+In+Proceedings+of+the+11th+Annual+Conference+on+Computational+Learning+Theory%2C+pages+209%E2%80%93217%2C+New+York%2C+N)
5. J. Jiang and D. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the International Conference on Research in Computational Linguistics, Taiwan. [find](https://scholar.google.com/scholar?q=J%2E+Jiang+and+D%2E+Conrath%2E+1997%2E+Semantic+similarity+based+on+corpus+statistics+and+lexical+taxonomy%2E+In+Proceedings+of+the+International+Conference+on+Research+in+Computational+Linguistics%2C+Taiwan%2E)
6. T. K. Landauer, P. Foltz, and D. Laham. 1998. Introduction to latent semantic analysis. Discourse Processes, 25. [find](https://scholar.google.com/scholar?q=T%2E+K%2E+Landauer%2C+P%2E+Foltz%2C+and+D%2E+Laham%2E+1998%2E+Introduction+to+latent+semantic+analysis%2E+Discourse+Processes%2C+25%2E)
7. C. Leacock and M. Chodorow. 1998. Combining local context and WordNet sense similiarity for word sense disambiguation. In WordNet, An Electronic Lexical Database. The MIT Press. [find](https://scholar.google.com/scholar?q=C%2E+Leacock+and+M%2E+Chodorow%2E+1998%2E+Combining+local+context+and+WordNet+sense+similiarity+for+word+sense+disambiguation%2E+In+WordNet%2C+An+Electronic+Lexical+Database%2E+The+MIT+Press%2E)
8. M.E. Lesk. 1986. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings of the SIGDOC Conference 1986, Toronto, June. [find](https://scholar.google.com/scholar?q=M%2EE%2E+Lesk%2E+1986%2E+Automatic+sense+disambiguation+using+machine+readable+dictionaries%3A+How+to+tell+a+pine+cone+from+an+ice+cream+cone%2E+In+Proceedings+of+the+SIGDOC+Conference+1986%2C+Toronto%2C+June%2E)
9. C.Y. Lin and E.H. Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of Human Language Technology Conference (HLT-NAACL 2003), Edmonton, Canada, May. [our card](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/)
10. D. Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison, WI. [find](https://scholar.google.com/scholar?q=D%2E+Lin%2E+1998%2E+An+information%2Dtheoretic+definition+of+similarity%2E+In+Proceedings+of+the+15th+International+Conference+on+Machine+Learning%2C+Madison%2C+WI%2E)
11. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), Philadelphia, PA, July. [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
12. S. Patwardhan, S. Banerjee, and T. Pedersen. 2003. Using measures of semantic relatedness for word sense disambiguation. In Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, Mexico City, February. [find](https://scholar.google.com/scholar?q=S%2E+Patwardhan%2C+S%2E+Banerjee%2C+and+T%2E+Pedersen%2E+2003%2E+Using+measures+of+semantic+relatedness+for+word+sense+disambiguation%2E+In+Proceedings+of+the+Fourth+International+Conference+on+Intelligent+Text+Proce)
13. P. Resnik. 1995. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, Canada. [find](https://scholar.google.com/scholar?q=P%2E+Resnik%2E+1995%2E+Using+information+content+to+evaluate+semantic+similarity%2E+In+Proceedings+of+the+14th+International+Joint+Conference+on+Artificial+Intelligence%2C+Montreal%2C+Canada%2E)
14. J. Rocchio, 1971. Relevance feedback in information retrieval. Prentice Hall, Ing. Englewood Cliffs, New Jersey. [find](https://scholar.google.com/scholar?q=J%2E+Rocchio%2C+1971%2E+Relevance+feedback+in+information+retrieval%2E+Prentice+Hall%2C+Ing%2E+Englewood+Cliffs%2C+New+Jersey%2E)
15. G. Salton and M.E. Lesk, 1971. Computer evaluation of indexing and text processing, pages 143–180. Prentice Hall, Ing. Englewood Cliffs, New Jersey. [find](https://scholar.google.com/scholar?q=G%2E+Salton+and+M%2EE%2E+Lesk%2C+1971%2E+Computer+evaluation+of+indexing+and+text+processing%2C+pages+143%E2%80%93180%2E+Prentice+Hall%2C+Ing%2E+Englewood+Cliffs%2C+New+Jersey%2E)
16. G. Salton, , and A. Bukley. 1997a. Term weighting approaches in automatic text retrieval. In Readings in Information Retrieval. Morgan Kaufmann Publishers, San Francisco, CA. [find](https://scholar.google.com/scholar?q=G%2E+Salton%2C+%2C+and+A%2E+Bukley%2E+1997a%2E+Term+weighting+approaches+in+automatic+text+retrieval%2E+In+Readings+in+Information+Retrieval%2E+Morgan+Kaufmann+Publishers%2C+San+Francisco%2C+CA%2E)
17. G. Salton, A. Singhal, M. Mitra, and C. Buckley. 1997b. Automatic text structuring and summarization. Information Processing and Management, 2(32). [find](https://scholar.google.com/scholar?q=G%2E+Salton%2C+A%2E+Singhal%2C+M%2E+Mitra%2C+and+C%2E+Buckley%2E+1997b%2E+Automatic+text+structuring+and+summarization%2E+Information+Processing+and+Management%2C+2%2832%29%2E)
18. K. Sparck-Jones. 1972. A statistical interpretation of term specificity and its applicatoin in retrieval. Journal of Documentation, 28(1):11–21. [find](https://scholar.google.com/scholar?q=K%2E+Sparck%2DJones%2E+1972%2E+A+statistical+interpretation+of+term+specificity+and+its+applicatoin+in+retrieval%2E+Journal+of+Documentation%2C+28%281%29%3A11%E2%80%9321%2E)
19. E. Voorhees. 1993. Using wordnet to disambiguate word senses for text retrieval. In Proceedings of the 16th annual international ACM SIGIR conference, Pittsburgh, PA. [find](https://scholar.google.com/scholar?q=E%2E+Voorhees%2E+1993%2E+Using+wordnet+to+disambiguate+word+senses+for+text+retrieval%2E+In+Proceedings+of+the+16th+annual+international+ACM+SIGIR+conference%2C+Pittsburgh%2C+PA%2E)
20. Z. Wu and M. Palmer. 1994. Verb semantics and lexical selection. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, Las Cruces, New Mexico. [our card](https://md.research.elamur.ai/free/en/papers/wu-1994-verb-semantics-lexical-selection/)
21. J. Xu and W. B. Croft. 1996. Query expansion using local and global document analysis. In Proceedings of the 19th annual international ACM SIGIR conference, Zurich, Switzerland. [find](https://scholar.google.com/scholar?q=J%2E+Xu+and+W%2E+B%2E+Croft%2E+1996%2E+Query+expansion+using+local+and+global+document+analysis%2E+In+Proceedings+of+the+19th+annual+international+ACM+SIGIR+conference%2C+Zurich%2C+Switzerland%2E)