---
title: "A Study on Similarity and Relatedness Using — Agirre 2009"
description: "The paper compares two families of methods for scoring how alike two words are: methods built on WordNet and methods learned from corpora."
authors: ["Eneko Agirre", "Enrique Alfonseca", "Keith Hall", "Jana Kravalova", "Marius Paşca", "Aitor Soroa"]
year: 2009
venue: "Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/N09-1003/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/) / A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches

# A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches

**Authors:** [Eneko Agirre](https://md.research.elamur.ai/free/en/authors/eneko-agirre/), [Enrique Alfonseca](https://md.research.elamur.ai/free/en/authors/enrique-alfonseca/), [Keith Hall](https://md.research.elamur.ai/free/en/authors/keith-hall/), Jana Kravalova, [Marius Paşca](https://md.research.elamur.ai/free/en/authors/marius-pasca/), [Aitor Soroa](https://md.research.elamur.ai/free/en/authors/aitor-soroa/) · **Year:** 2009 · **Venue:** Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N09-1003/) · [PDF](https://aclanthology.org/N09-1003.pdf) · [OpenAlex](https://openalex.org/W2170682101)

**Citations:** 860 citations (openalex-snapshot · 2026-10-14) · [257](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper compares two families of methods for scoring how alike two words are: methods built on WordNet and methods learned from corpora. It also examines how they differ on *similarity* data versus *relatedness* data. The WordNet side runs personalized PageRank over a graph of synsets and words, once for each word. The two resulting probability distributions are then compared with cosine. The distributional side builds vector space models from a crawled Web corpus of about 1.6 Terawords, using bag-of-words, context windows, or syntactic dependency templates, with χ² weighting and cosine. Each approach is the strongest in its own class on RG and WordSim353. An SVM that combines them reaches the best published scores on both datasets. The authors also run what they describe as the first cross-lingual (English–Spanish) evaluation of such measures, with only small losses. This matters because it shows Web-scale distributional statistics can match curated lexical resources, and because the two families turn out to be complementary.

## Key points
- WordNet graphs used: MCR16 (Multilingual Central Repository with WordNet 1.6), WN30, and WN30g (WordNet 3.0 plus disambiguated gloss relations). PageRank used the default damping of 0.85 and 30 iterations, with no tuning.
- On WordSim353, WN30g scores 0.66 Spearman, compared with 0.53 for MCR16 and 0.56 for WN30. The authors credit the disambiguated glosses for most of this gain, mainly because they model relatedness better.
- The Web corpus started from about four billion English documents crawled in August 2008. Computing all test-pair similarities with MapReduce takes about 15 minutes on 2,000 cores.
- Context windows suit similarity best: a window of 4 gives 0.89 on RG. Bag-of-words suits relatedness best.
- A supervised SVM combination, run in Weka with 10-fold cross-validation, reaches 0.78 Spearman on WordSim353. This is above Gabrilovich and Markovitch's Wikipedia-based 0.75, and the improvement on RG is statistically significant.

## Interesting ideas
- **Cross-lingual similarity through a shared concept graph:** Wordnets aligned via an InterLingual Index let words from two languages be mapped onto one set of concepts. A Ukrainian–English word pair could therefore be compared on a common graph, provided an aligned Ukrainian wordnet exists.
- **Personalized PageRank as a word signature:** Starting all random-jump mass at the target word yields a distribution over concepts. This captures paths through the graph, not just shortest distances, and the idea carries over to any knowledge graph.
- **Distributional fallback for missing words:** Corpus-based similarity was used to cover items absent from WordNet, which gave the best unsupervised results. A curated resource can be extended cheaply in the same way.
- **Report confidence intervals and use Spearman:** The authors report 95% intervals and show that many published RG/WordSim353 differences cannot be told apart. They prefer Spearman because it needs no dataset-specific output mapping. Both habits would improve evaluation of any small benchmark.
- **Separate similarity from relatedness:** Different context types serve each notion better, so the window or feature design should match the target task, for example synonym finding versus topical association.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

257 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 240 not yet.

- [Random Walks for Knowledge-Based Word Sense Disambiguation](https://md.research.elamur.ai/free/en/papers/agirre-2014-random-walks-knowledge-based-word/)
- [Massively Multilingual Word Embeddings](https://md.research.elamur.ai/free/en/papers/ammar-2016-massively-multilingual-word-embeddings/)
- [How we BLESSed distributional semantic evaluation](https://md.research.elamur.ai/free/en/papers/baroni-2011-how-we-blessed-distributional-semantic/)
- [Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors](https://md.research.elamur.ai/free/en/papers/baroni-2014-don-t-count-predict-systematic/)
- [Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings](https://md.research.elamur.ai/free/en/papers/bommasani-2020-interpreting-pretrained-contextualized-representations-reductions/)
- [Improving Vector Space Word Representations Using Multilingual Correlation](https://md.research.elamur.ai/free/en/papers/faruqui-2014-improving-vector-space-word-representations/)
- [Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://md.research.elamur.ai/free/en/papers/faruqui-2016-problems-evaluation-word-embeddings-word/)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://md.research.elamur.ai/free/en/papers/gerz-2016-simverb-3500-large-scale-evaluation/)
- [SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
- [SensEmbed: Learning Sense Embeddings for Word and Relational Similarity](https://md.research.elamur.ai/free/en/papers/iacobacci-2015-sensembed-learning-sense-embeddings-word/)
- [Inducing Crosslingual Distributed Representations of Words](https://md.research.elamur.ai/free/en/papers/klementiev-2012-inducing-crosslingual-distributed-representations-words/)
- [Automatic Evaluation of Topic Coherence](https://md.research.elamur.ai/free/en/papers/newman-2010-automatic-evaluation-topic-coherence/)
- [Abstract Syntax as Interlingua: Scaling Up the Grammatical Framework from Controlled Languages to Robust Pipelines](https://md.research.elamur.ai/free/en/papers/ranta-2020-abstract-syntax-as-interlingua-scaling/)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://md.research.elamur.ai/free/en/papers/ravfogel-2020-null-it-out-guarding-protected/)
- [Multi-Prototype Vector-Space Models of Word Meaning](https://md.research.elamur.ai/free/en/papers/reisinger-2010-multi-prototype-vector-space-models/)
- [From Paraphrase Database to Compositional Paraphrase Model and Back](https://md.research.elamur.ai/free/en/papers/wieting-2015-paraphrase-database-compositional-paraphrase-model/)
- [Learning Discriminative Projections for Text Similarity Measures](https://md.research.elamur.ai/free/en/papers/yih-2011-learning-discriminative-projections-text-similarity/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{agirre2009study,
  title = {{A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches}},
  author = {Agirre, Eneko and Alfonseca, Enrique and Hall, Keith and Kravalova, Jana and Paşca, Marius and Soroa, Aitor},
  booktitle = {Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
  doi = {10.3115/1620754.1620758},
  url = {https://doi.org/10.3115/1620754.1620758}
}
```

- [← previous A Scalable Global Model for Summarization](https://md.research.elamur.ai/free/en/papers/gillick-2009-scalable-global-model-summarization/)
- [next → A Survey of Paraphrasing and Textual Entailment Methods](https://md.research.elamur.ai/free/en/papers/androutsopoulos-2009-survey-paraphrasing-textual-entailment-methods/)
- [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/)

## References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. E. Agirre and A. Soroa. 2009. Personalizing pagerank for word sense disambiguation. In Proc. of EACL 2009, Athens, Greece. [find](https://scholar.google.com/scholar?q=E%2E+Agirre+and+A%2E+Soroa%2E+2009%2E+Personalizing+pagerank+for+word+sense+disambiguation%2E+In+Proc%2E+of+EACL+2009%2C+Athens%2C+Greece%2E)
2. M.A. Alvarez and S.J. Lim. 2007. A Graph Modeling of Semantic Similarity between Words. Proc. of the Conference on Semantic Computing, pages 355–362. [find](https://scholar.google.com/scholar?q=M%2EA%2E+Alvarez+and+S%2EJ%2E+Lim%2E+2007%2E+A+Graph+Modeling+of+Semantic+Similarity+between+Words%2E+Proc%2E+of+the+Conference+on+Semantic+Computing%2C+pages+355%E2%80%93362%2E)
3. J. Atserias, L. Villarejo, G. Rigau, E. Agirre, J. Carroll, B. Magnini, and P. Vossen. 2004. The meaning multilingual central repository. In Proc. of Global WordNet Conference, Brno, Czech Republic. [find](https://scholar.google.com/scholar?q=J%2E+Atserias%2C+L%2E+Villarejo%2C+G%2E+Rigau%2C+E%2E+Agirre%2C+J%2E+Carroll%2C+B%2E+Magnini%2C+and+P%2E+Vossen%2E+2004%2E+The+meaning+multilingual+central+repository%2E+In+Proc%2E+of+Global+WordNet+Conference%2C+Brno%2C+Czech+Republic%2E)
4. D. Bollegala, Matsuo Y., and M. Ishizuka. 2007. Measuring semantic similarity between words using web search engines. In Proceedings of WWW’2007. [find](https://scholar.google.com/scholar?q=D%2E+Bollegala%2C+Matsuo+Y%2E%2C+and+M%2E+Ishizuka%2E+2007%2E+Measuring+semantic+similarity+between+words+using+web+search+engines%2E+In+Proceedings+of+WWW%E2%80%992007%2E)
5. A. Budanitsky and G. Hirst. 2006. Evaluating WordNet-based Measures of Lexical Semantic Relatedness. Computational Linguistics, 32(1):13–47. [find](https://scholar.google.com/scholar?q=A%2E+Budanitsky+and+G%2E+Hirst%2E+2006%2E+Evaluating+WordNet%2Dbased+Measures+of+Lexical+Semantic+Relatedness%2E+Computational+Linguistics%2C+32%281%29%3A13%E2%80%9347%2E) [our card](https://md.research.elamur.ai/free/en/papers/budanitsky-2006-evaluating-wordnet-based-measures-lexical/)
6. H. Chen, M. Lin, and Y. Wei. 2006. Novel association measures using web search with double checking. In Proceedings of COCLING/ACL 2006. [find](https://scholar.google.com/scholar?q=H%2E+Chen%2C+M%2E+Lin%2C+and+Y%2E+Wei%2E+2006%2E+Novel+association+measures+using+web+search+with+double+checking%2E+In+Proceedings+of+COCLING%2FACL+2006%2E)
7. J. Daude, L. Padro, and G. Rigau. 2000. Mapping Word-Nets using structural information. In Proceedings of ACL’2000, Hong Kong. [find](https://scholar.google.com/scholar?q=J%2E+Daude%2C+L%2E+Padro%2C+and+G%2E+Rigau%2E+2000%2E+Mapping+Word%2DNets+using+structural+information%2E+In+Proceedings+of+ACL%E2%80%992000%2C+Hong+Kong%2E)
8. C. Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database and Some of its Applications. MIT Press, Cambridge, Mass. [find](https://scholar.google.com/scholar?q=C%2E+Fellbaum%2C+editor%2E+1998%2E+WordNet%3A+An+Electronic+Lexical+Database+and+Some+of+its+Applications%2E+MIT+Press%2C+Cambridge%2C+Mass%2E)
9. L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin. 2002. Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116–131. [find](https://scholar.google.com/scholar?q=L%2E+Finkelstein%2C+E%2E+Gabrilovich%2C+Y%2E+Matias%2C+E%2E+Rivlin%2C+Z%2E+Solan%2C+G%2E+Wolfman%2C+and+E%2E+Ruppin%2E+2002%2E+Placing+Search+in+Context%3A+The+Concept+Revisited%2E+ACM+Transactions+on+Information+Systems%2C+20%281%29%3A116%E2%80%9313)
10. E. Gabrilovich and S. Markovitch. 2007. Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis. Proc of IJCAI, pages 6–12. [find](https://scholar.google.com/scholar?q=E%2E+Gabrilovich+and+S%2E+Markovitch%2E+2007%2E+Computing+Semantic+Relatedness+using+Wikipedia%2Dbased+Explicit+Semantic+Analysis%2E+Proc+of+IJCAI%2C+pages+6%E2%80%9312%2E)
11. T. H. Haveliwala. 2002. Topic-sensitive pagerank. In WWW ’02: Proceedings of the 11th international conference on World Wide Web, pages 517–526. [find](https://scholar.google.com/scholar?q=T%2E+H%2E+Haveliwala%2E+2002%2E+Topic%2Dsensitive+pagerank%2E+In+WWW+%E2%80%9902%3A+Proceedings+of+the+11th+international+conference+on+World+Wide+Web%2C+pages+517%E2%80%93526%2E)
12. T. Hughes and D. Ramage. 2007. Lexical semantic relatedness with random graph walks. In Proceedings of EMNLP-CoNLL-2007, pages 581–589. [find](https://scholar.google.com/scholar?q=T%2E+Hughes+and+D%2E+Ramage%2E+2007%2E+Lexical+semantic+relatedness+with+random+graph+walks%2E+In+Proceedings+of+EMNLP%2DCoNLL%2D2007%2C+pages+581%E2%80%93589%2E)
13. M. Jarmasz. 2003. Roget’s Thesuarus as a lexical resource for Natural Language Processing. [find](https://scholar.google.com/scholar?q=M%2E+Jarmasz%2E+2003%2E+Roget%E2%80%99s+Thesuarus+as+a+lexical+resource+for+Natural+Language+Processing%2E)
14. J.J. Jiang and D.W. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics, volume 33. Taiwan. [find](https://scholar.google.com/scholar?q=J%2EJ%2E+Jiang+and+D%2EW%2E+Conrath%2E+1997%2E+Semantic+similarity+based+on+corpus+statistics+and+lexical+taxonomy%2E+In+Proceedings+of+International+Conference+on+Research+in+Computational+Linguistics%2C+volume+33%2E+)
15. C. Leacock and M. Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. WordNet: An Electronic Lexical Database, 49(2):265–283. [find](https://scholar.google.com/scholar?q=C%2E+Leacock+and+M%2E+Chodorow%2E+1998%2E+Combining+local+context+and+WordNet+similarity+for+word+sense+identification%2E+WordNet%3A+An+Electronic+Lexical+Database%2C+49%282%29%3A265%E2%80%93283%2E)
16. D. Lin. 1998a. An information-theoretic definition of similarity. In Proc. of ICML, pages 296–304, Wisconsin, USA. [find](https://scholar.google.com/scholar?q=D%2E+Lin%2E+1998a%2E+An+information%2Dtheoretic+definition+of+similarity%2E+In+Proc%2E+of+ICML%2C+pages+296%E2%80%93304%2C+Wisconsin%2C+USA%2E)
17. D. Lin. 1998b. Automatic Retrieval and Clustering of Similar Words. In Proceedings of ACL-98. [find](https://scholar.google.com/scholar?q=D%2E+Lin%2E+1998b%2E+Automatic+Retrieval+and+Clustering+of+Similar+Words%2E+In+Proceedings+of+ACL%2D98%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-1998-automatic-retrieval-clustering-similar-words/)
18. G.A. Miller and W.G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. [find](https://scholar.google.com/scholar?q=G%2EA%2E+Miller+and+W%2EG%2E+Charles%2E+1991%2E+Contextual+correlates+of+semantic+similarity%2E+Language+and+Cognitive+Processes%2C+6%281%29%3A1%E2%80%9328%2E)
19. J. Nivre. 2006. Inductive Dependency Parsing, volume 34 of Text, Speech and Language Technology. Springer. [find](https://scholar.google.com/scholar?q=J%2E+Nivre%2E+2006%2E+Inductive+Dependency+Parsing%2C+volume+34+of+Text%2C+Speech+and+Language+Technology%2E+Springer%2E)
20. S. Pado and M. Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. [find](https://scholar.google.com/scholar?q=S%2E+Pado+and+M%2E+Lapata%2E+2007%2E+Dependency%2Dbased+construction+of+semantic+space+models%2E+Computational+Linguistics%2C+33%282%29%3A161%E2%80%93199%2E) [our card](https://md.research.elamur.ai/free/en/papers/pado-2007-dependency-based-construction-semantic-space/)
21. S. Patwardhan and T. Pedersen. 2006. Using WordNet-based Context Vectors to Estimate the Semantic Relatedness of Concepts. In Proceedings of the EACL Workshop on Making Sense of Sense: Bringing Computational Linguistics and Pycholinguistics Together, pages 1–8, Trento, Italy. [find](https://scholar.google.com/scholar?q=S%2E+Patwardhan+and+T%2E+Pedersen%2E+2006%2E+Using+WordNet%2Dbased+Context+Vectors+to+Estimate+the+Semantic+Relatedness+of+Concepts%2E+In+Proceedings+of+the+EACL+Workshop+on+Making+Sense+of+Sense%3A+Bringing+Comput)
22. P. Resnik. 1995. Using Information Content to Evaluate Semantic Similarity in a Taxonomy. Proc. of IJCAI, 14:448–453. [find](https://scholar.google.com/scholar?q=P%2E+Resnik%2E+1995%2E+Using+Information+Content+to+Evaluate+Semantic+Similarity+in+a+Taxonomy%2E+Proc%2E+of+IJCAI%2C+14%3A448%E2%80%93453%2E) [our card](https://md.research.elamur.ai/free/en/papers/resnik-1995-information-content-evaluate-semantic-similarity/)
23. H. Rubenstein and J.B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. [find](https://scholar.google.com/scholar?q=H%2E+Rubenstein+and+J%2EB%2E+Goodenough%2E+1965%2E+Contextual+correlates+of+synonymy%2E+Communications+of+the+ACM%2C+8%2810%29%3A627%E2%80%93633%2E)
24. M Ruiz-Casado, E. Alfonseca, and P. Castells. 2005. Using context-window overlapping in Synonym Discovery and Ontology Extension. In Proceedings of RANLP-2005, Borovets, Bulgaria,. [find](https://scholar.google.com/scholar?q=M+Ruiz%2DCasado%2C+E%2E+Alfonseca%2C+and+P%2E+Castells%2E+2005%2E+Using+context%2Dwindow+overlapping+in+Synonym+Discovery+and+Ontology+Extension%2E+In+Proceedings+of+RANLP%2D2005%2C+Borovets%2C+Bulgaria%2C%2E)
25. M. Sahami and T.D. Heilman. 2006. A web-based kernel function for measuring the similarity of short text snippets. Proc. of WWW, pages 377–386. [find](https://scholar.google.com/scholar?q=M%2E+Sahami+and+T%2ED%2E+Heilman%2E+2006%2E+A+web%2Dbased+kernel+function+for+measuring+the+similarity+of+short+text+snippets%2E+Proc%2E+of+WWW%2C+pages+377%E2%80%93386%2E)
26. M. Strube and S.P. Ponzetto. 2006. WikiRelate! Computing Semantic Relatedness Using Wikipedia. In Proceedings of the AAAI-2006, pages 1419–1424. [find](https://scholar.google.com/scholar?q=M%2E+Strube+and+S%2EP%2E+Ponzetto%2E+2006%2E+WikiRelate%21+Computing+Semantic+Relatedness+Using+Wikipedia%2E+In+Proceedings+of+the+AAAI%2D2006%2C+pages+1419%E2%80%931424%2E)
27. P.D. Turney. 2001. Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL. Lecture Notes in Computer Science, 2167:491–502. [find](https://scholar.google.com/scholar?q=P%2ED%2E+Turney%2E+2001%2E+Mining+the+Web+for+Synonyms%3A+PMI%2DIR+versus+LSA+on+TOEFL%2E+Lecture+Notes+in+Computer+Science%2C+2167%3A491%E2%80%93502%2E)
28. P. Vossen, editor. 1998. EuroWordNet: A Multilingual Database with Lexical Semantic Networks. Kluwer Academic Publishers. [find](https://scholar.google.com/scholar?q=P%2E+Vossen%2C+editor%2E+1998%2E+EuroWordNet%3A+A+Multilingual+Database+with+Lexical+Semantic+Networks%2E+Kluwer+Academic+Publishers%2E)
29. Z. Wu and M. Palmer. 1994. Verb semantics and lexical selection. In Proc. of ACL, pages 133–138, Las Cruces, New Mexico. [find](https://scholar.google.com/scholar?q=Z%2E+Wu+and+M%2E+Palmer%2E+1994%2E+Verb+semantics+and+lexical+selection%2E+In+Proc%2E+of+ACL%2C+pages+133%E2%80%93138%2C+Las+Cruces%2C+New+Mexico%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-1994-verb-semantics-lexical-selection/)
30. D. Yang and D.M.W. Powers. 2005. Measuring semantic similarity in the taxonomy of WordNet. Proceedings of the Australasian conference on Computer Science. [find](https://scholar.google.com/scholar?q=D%2E+Yang+and+D%2EM%2EW%2E+Powers%2E+2005%2E+Measuring+semantic+similarity+in+the+taxonomy+of+WordNet%2E+Proceedings+of+the+Australasian+conference+on+Computer+Science%2E)