---
title: "SemEval-2012 Task 2: Measuring Degrees of Relational Similarity"
description: "Most earlier work on semantic relations, including previous SemEval tasks, treats relations as discrete… 22 references, cited by 77 papers here."
authors: ["David Jurgens", "Saif Mohammad", "Peter Turney", "Keith Holyoak"]
year: 2012
venue: "*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)"
source: "https://aclanthology.org/S12-1047/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jurgens-2012-semeval-2012-task-2-measuring/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jurgens-2012-semeval-2012-task-2-measuring/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/) / SemEval-2012 Task 2: Measuring Degrees of Relational Similarity

# SemEval-2012 Task 2: Measuring Degrees of Relational Similarity

**Authors:** [David Jurgens](https://md.research.elamur.ai/free/en/authors/david-jurgens/), [Saif Mohammad](https://md.research.elamur.ai/free/en/authors/saif-mohammad/), [Peter Turney](https://md.research.elamur.ai/free/en/authors/peter-turney/) and Keith Holyoak · **Year:** 2012 · **Venue:** [\*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)](https://md.research.elamur.ai/free/en/venues/starsem/) ([\*SEM 2012](https://md.research.elamur.ai/free/en/venues/starsem/2012/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S12-1047/) · [PDF](https://aclanthology.org/S12-1047.pdf) · [OpenAlex](https://openalex.org/W2098801107)

**Citations:** 182 citations (OpenAlex, 2026-10-14) · [77](https://md.research.elamur.ai/free/en/papers/jurgens-2012-semeval-2012-task-2-measuring/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most earlier work on semantic relations, including previous SemEval tasks, treats relations as discrete classes: a word pair either belongs to CAUSE : EFFECT or it does not. This task instead asks how *typical* a pair is of its relation. For example, dog:bark is a better example of ENTITY : SOUND than floor:squeak. Systems were given a few paradigm pairs and a schematic description for each relation. They then had to rank other pairs by how prototypical they are.

The organizers built the first graded relational-similarity dataset, covering 79 subcategories from the Bejar et al. (1991) taxonomy. It was collected with Amazon Mechanical Turk in two phases: Turkers first generated new example pairs, then judged how representative each pair was. Three teams submitted six systems, which were scored against the Turker rankings in two ways.

Results were modest overall. A simple PMI baseline beat every system except UTD-NB, and no approach did well across all relation types. The work matters for tasks that need graded judgments, such as ranked relational search and choosing the best source analogy.

## Key points
- **Relation inventory:** the taxonomy has 10 high-level categories (e.g., CAUSE-PURPOSE, SPACE-TIME) and 79 subcategories. Earlier relation tasks used only 7 or 9 relation types.
- **Data collection:** Turkers produced 3160 pairs, of which 2905 were unique and 2823 remained after spelling cleanup. Another 395 pairs were added by reversing the arguments of sampled pairs, for 3218 rated pairs in total.
- **Quality control:** roughly 7% of Turker responses were discarded because the worker chose the wrong relation description in a multiple-choice check.
- **Results (average Spearman's ρ with reversed pairs included):** UTD-NB 0.229, UTD-SVM 0.116, PMI baseline 0.112, Duluth systems about 0.04–0.05, BUAP 0.014, random 0.018. Removing reversed pairs lowered the scores of the order-aware systems, which suggests those systems were correctly ranking reversals low.
- **Difficulty by category:** systems did best on the Similar category and worst on Non-Attribute relations. Difficulty did not correlate with the spread of prototypicality scores within a subcategory.

## Interesting ideas
- **Defining a class by examples plus a schema.** Each relation is specified only by a handful of seed examples and a short template such as "X is the goal of Y". This setup is useful wherever categories must be specified with almost no labeled data, for instance in low-resource Ukrainian annotation projects.
- **A gate question before crowdsourced generation.** Workers first pick the correct relation description from distractors, and only those who pass may contribute examples. This cheap filter removes confused or careless workers and could be reused in any crowdsourcing or volunteer annotation pipeline.
- **Reversed pairs as built-in probes.** Adding pairs with swapped arguments tests whether a model is sensitive to the direction of a relation. The authors also measure the rank gap between a pair and its reversal (RMSE) as a separate diagnostic, a trick that transfers to evaluating embeddings or knowledge-graph models.
- **Testing a simple association baseline.** PMI did surprisingly well because every pair within a subcategory is already related. Including such a baseline exposes whether the task design accidentally rewards plain co-occurrence instead of relational understanding.
- **Choosing a metric that allows ablation.** Rank correlation lets single items be removed for post-hoc analysis without disturbing the rest of the evaluation, whereas grouped comparison questions do not. This is worth considering when deciding how an evaluation should be structured.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Research Council Canada](https://md.research.elamur.ai/free/en/government/national-research-council-canada/) — Government body, Ottawa, Canada · 40 works · 20,943 citations
- [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) — University, Los Angeles, United States · 54 works · 53,688 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 463 works

## Cite

Cite as: David Jurgens, Saif Mohammad, Peter Turney and Keith Holyoak. 2012. SemEval-2012 Task 2: Measuring Degrees of Relational Similarity. \*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jurgens2012semeval,
  title = {{SemEval-2012 Task 2: Measuring Degrees of Relational Similarity}},
  author = {David Jurgens, Saif Mohammad, Peter Turney and Keith Holyoak},
  booktitle = {*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)},
  year = {2012},
  url = {https://aclanthology.org/S12-1047/}
}
```

- [← previous TakeLab: Systems for Measuring Semantic Text Similarity](https://md.research.elamur.ai/free/en/papers/saric-2012-takelab-systems-measuring-semantic-text/)
- [next → HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://md.research.elamur.ai/free/en/papers/dale-2012-hoo-2012-report-preposition-determiner/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/)

## References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Isaac I. Bejar, Roger Chaffin, and Susan E. Embretson. 1991. Cognitive and Psychometric Analysis of Analogical Problem Solving. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Isaac+I%2E+Bejar%2C+Roger+Chaffin%2C+and+Susan+E%2E+Embretson%2E+1991%2E+Cognitive+and+Psychometric+Analysis+of+Analogical+Problem+Solving%2E+Springer%2DVerlag%2E)
2. Cristina Butnariu, Su Nam Kim, Preslav Nakov, Dair-muid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2010. Semeval-2010 task 9: The interpretation of noun compounds using paraphrasing verbs and prepositions. In Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval-2010), pa… [find](https://scholar.google.com/scholar?q=Cristina+Butnariu%2C+Su+Nam+Kim%2C+Preslav+Nakov%2C+Dair%2Dmuid+O%CC%81+Se%CC%81aghdha%2C+Stan+Szpakowicz%2C+and+Tony+Veale%2E+2010%2E+Semeval%2D2010+task+9%3A+The+interpretation+of+noun+compounds+using+paraphrasing+verbs+and+prep)
3. Michael J. Cafarella, Michele Banko, and Oren Etzioni. 2006. Relational web search. In WWW Conference. [find](https://scholar.google.com/scholar?q=Michael+J%2E+Cafarella%2C+Michele+Banko%2C+and+Oren+Etzioni%2E+2006%2E+Relational+web+search%2E+In+WWW+Conference%2E)
4. Kenneth W. Church and Patrick Hanks. 1990. [Word association norms, mutual information, and lexicography](https://md.research.elamur.ai/free/en/papers/church-1990-word-association-norms-mutual-information/). Computational linguistics, 16(1):22–29.
5. Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2):155– 170. [find](https://scholar.google.com/scholar?q=Dedre+Gentner%2E+1983%2E+Structure%2Dmapping%3A+A+theoretical+framework+for+analogy%2E+Cognitive+science%2C+7%282%29%3A155%E2%80%93+170%2E)
6. Roxana Girju, Preslav Nakov, Vivi Nastase, Stan Szpakowicz, Peter Turney, and Deniz Yuret. 2007. [SemEval-2007 task 04: Classification of semantic relations between nominals](https://md.research.elamur.ai/free/en/papers/girju-2007-semeval-2007-task-04-classification/). In Proceedings of the 4th International Workshop on Semantic Evaluation (SemEval-2007).
7. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. [SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/). In Proceedings of the 5th International…
8. Keith J. Holyoak. 2012. Analogy and relational reasoning. In Oxford handbook of thinking and reasoning, pages 234–259. Oxford University Press. [find](https://scholar.google.com/scholar?q=Keith+J%2E+Holyoak%2E+2012%2E+Analogy+and+relational+reasoning%2E+In+Oxford+handbook+of+thinking+and+reasoning%2C+pages+234%E2%80%93259%2E+Oxford+University+Press%2E)
9. Jordan J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. Working Paper. [find](https://scholar.google.com/scholar?q=Jordan+J%2E+Louviere%2E+1991%2E+Best%2Dworst+scaling%3A+A+model+for+the+largest+difference+judgments%2E+Working+Paper%2E)
10. Saif M. Mohammad and Peter D. Turney. 2010. [Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon](https://md.research.elamur.ai/free/en/papers/mohammad-2010-emotions-evoked-common-words-phrases/). In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 26–34. Association for Computati…
11. Preslav Nakov and Marti A. Hearst. 2008. Solving relational similarity problems using the web as a corpus. In Proceedings of ACL, volume 8. [find](https://scholar.google.com/scholar?q=Preslav+Nakov+and+Marti+A%2E+Hearst%2E+2008%2E+Solving+relational+similarity+problems+using+the+web+as+a+corpus%2E+In+Proceedings+of+ACL%2C+volume+8%2E)
12. Vivi Nastase and Stan Szpakowicz. 2003. Exploring noun-modifier semantic relations. In Fifth International Workshop on Computational Semantics (IWCS-5), pages 285–301. ACL Press Tilburg,, The Netherlands. [find](https://scholar.google.com/scholar?q=Vivi+Nastase+and+Stan+Szpakowicz%2E+2003%2E+Exploring+noun%2Dmodifier+semantic+relations%2E+In+Fifth+International+Workshop+on+Computational+Semantics+%28IWCS%2D5%29%2C+pages+285%E2%80%93301%2E+ACL+Press+Tilburg%2C%2C+The+Netherla)
13. Vivi Nastase, Jelber Sayyad-Shirabad, Marina Sokolova, and Stan Szpakowicz. 2006. Learning noun-modifier semantic relations with corpus-based and wordnet-based features. In Proceedings of AAAI, volume 21, page 781. [find](https://scholar.google.com/scholar?q=Vivi+Nastase%2C+Jelber+Sayyad%2DShirabad%2C+Marina+Sokolova%2C+and+Stan+Szpakowicz%2E+2006%2E+Learning+noun%2Dmodifier+semantic+relations+with+corpus%2Dbased+and+wordnet%2Dbased+features%2E+In+Proceedings+of+AAAI%2C+volume)
14. Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and hb. [find](https://scholar.google.com/scholar?q=Bryan+Orme%2E+2009%2E+Maxdiff+analysis%3A+Simple+counting%2C+individual%2Dlevel+logit%2C+and+hb%2E)
15. Patrick Pantel and Marco Pennacchiotti. 2006. [Espresso: leveraging generic patterns for automatically harvesting semantic relations](https://md.research.elamur.ai/free/en/papers/pantel-2006-espresso-leveraging-generic-patterns-automatically/). In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 11…
16. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. [Cheap and fast—but is it good?](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/) Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263. Association for Computational Lingui…
17. M. Stephens, M. Palakal, S. Mukhopadhyay, R. Raje, J. Mostafa, et al. 2001. Detecting gene relations from medline abstracts. In Pacific Symposium on Biocomputing, volume 6, pages 483–495. Citeseer. [find](https://scholar.google.com/scholar?q=M%2E+Stephens%2C+M%2E+Palakal%2C+S%2E+Mukhopadhyay%2C+R%2E+Raje%2C+J%2E+Mostafa%2C+et+al%2E+2001%2E+Detecting+gene+relations+from+medline+abstracts%2E+In+Pacific+Symposium+on+Biocomputing%2C+volume+6%2C+pages+483%E2%80%93495%2E+Citeseer%2E)
18. Peter D. Turney and Michael L Littman. 2005. Corpus-based learning of analogies and semantic relations. Machine Learning, 60(1–3):251–278. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney+and+Michael+L+Littman%2E+2005%2E+Corpus%2Dbased+learning+of+analogies+and+semantic+relations%2E+Machine+Learning%2C+60%281%E2%80%933%29%3A251%E2%80%93278%2E)
19. Peter D. Turney. 2001. Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. In Proceedings of the Twelfth European Conference on Machine Learning (ECML-2001), pages 491–502. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney%2E+2001%2E+Mining+the+Web+for+synonyms%3A+PMI%2DIR+versus+LSA+on+TOEFL%2E+In+Proceedings+of+the+Twelfth+European+Conference+on+Machine+Learning+%28ECML%2D2001%29%2C+pages+491%E2%80%93502%2E)
20. Peter D. Turney. 2005. Measuring semantic similarity by latent relational analysis. In Proceedings of IJCAI, pages 1136–1141. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney%2E+2005%2E+Measuring+semantic+similarity+by+latent+relational+analysis%2E+In+Proceedings+of+IJCAI%2C+pages+1136%E2%80%931141%2E)
21. Peter D. Turney. 2006. [Similarity of semantic relations](https://md.research.elamur.ai/free/en/papers/turney-2006-similarity-semantic-relations/). Computational Linguistics, 32(3):379–416.
22. Peter D. Turney. 2008. The latent relation mapping engine: Algorithm and experiments. Journal of Artificial Intelligence Research, 33(1):615–655. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney%2E+2008%2E+The+latent+relation+mapping+engine%3A+Algorithm+and+experiments%2E+Journal+of+Artificial+Intelligence+Research%2C+33%281%29%3A615%E2%80%93655%2E)

*Updated 24 October 2026*