---
title: "SemEval-2007 Task 04 — paper by Girju et al. (2007)"
description: "This SemEval-2007 task set up a shared benchmark for deciding which semantic… 76.3% accuracy. 15 references, cited by 76 papers here."
authors: ["Roxana Girju", "Preslav Nakov", "Vivi Nastase", "Stan Szpakowicz", "Peter Turney", "Deniz Yuret"]
year: 2007
venue: "Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)"
source: "https://aclanthology.org/S07-1003/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/girju-2007-semeval-2007-task-04-classification/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/girju-2007-semeval-2007-task-04-classification/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/) / SemEval-2007 Task 04: Classification of Semantic Relations between Nominals

# SemEval-2007 Task 04: Classification of Semantic Relations between Nominals

**Authors:** [Roxana Girju](https://md.research.elamur.ai/free/en/authors/roxana-girju/), [Preslav Nakov](https://md.research.elamur.ai/free/en/authors/preslav-nakov/), [Vivi Nastase](https://md.research.elamur.ai/free/en/authors/vivi-nastase/), [Stan Szpakowicz](https://md.research.elamur.ai/free/en/authors/stan-szpakowicz/), [Peter Turney](https://md.research.elamur.ai/free/en/authors/peter-turney/) and [Deniz Yuret](https://md.research.elamur.ai/free/en/authors/deniz-yuret/) · **Year:** 2007 · **Venue:** [Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)](https://md.research.elamur.ai/free/en/venues/international-workshop-on-semantic-evaluations/) ([International Workshop on Semantic Evaluations 2007](https://md.research.elamur.ai/free/en/venues/international-workshop-on-semantic-evaluations/2007/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S07-1003/) · [PDF](https://aclanthology.org/S07-1003.pdf) · [OpenAlex](https://openalex.org/W7061918868)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [76](https://md.research.elamur.ai/free/en/papers/girju-2007-semeval-2007-task-04-classification/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This SemEval-2007 task set up a shared benchmark for deciding which semantic relation holds between two nominals (nouns or base noun phrases, not named entities) marked in an English sentence. The organizers did not try to impose one universal relation inventory. They picked seven relations that recur across many existing schemes and turned each into its own binary classification problem: Cause-Effect, Instrument-Agency, Product-Producer, Origin-Entity, Theme-Tool, Part-Whole and Content-Container. Example sentences came from manually designed Google wildcard patterns. The tagged nominals were annotated with WordNet 3.0 senses and the query used to find them, then labeled true or false by two annotators who settled every disagreement by discussion. Fourteen teams entered 15 systems, grouped by whether they used the WordNet sense labels and/or the query. The best system reached 76.3% macro-averaged accuracy. Supplying WordNet senses and adding training data gave significant gains, while knowing the query did not. The resulting freely available data set gave the field a common testbed for relation classification.

## Key points
- Seven relations, each with 140 training examples and at least 70 test examples (71–93 per relation). The Content-Container training set was made available early for development purposes, two months ahead of the test set's release.
- Before consensus, annotators agreed on 70.3% of relation labels and 71.9% of WordNet sense labels on average. Agreement on Theme-Tool was only 47.8%.
- Systems fell into four categories: A (neither WordNet labels nor query), B (WordNet only), C (query only), D (both). The top result came from UIUC in category B: 76.3% accuracy, F = 72.4.
- Using the WordNet sense labels raised average F by 6.1 points (significant). Going from 35 to 140 training examples raised it by 8.3 points. The query gave an average gain of only 0.9, which was not significant.
- Difficulty differed by relation. The best F was 83.7 for Product-Producer and 68.6 for Origin-Entity. Origin-Entity and Theme-Tool had the worst average rank across systems.

## Interesting ideas
- **Binary tasks instead of one taxonomy:** each relation is a separate yes/no task, so the benchmark avoids arguing over one universal scheme. This makes it easy to add relations or reuse individual ones in other projects, including a Ukrainian relation data set.
- **Pattern-based retrieval that yields near-miss negatives:** the same search patterns return both true instances and look-alikes that fail the relation. These hard negatives teach a classifier more than random negatives, and the trick suits any low-resource annotation effort.
- **Releasing optional information and splitting systems by what they used:** WordNet senses and the source query were shipped with the data, and results were reported separately by input category. This lets organizers measure how much each information source actually helps, a useful design for shared tasks.
- **Learning curves from nested training subsets:** scores were collected at 35, 70, 105 and 140 training examples and the differences were tested for significance. This shows cheaply whether more annotation is worth paying for.
- **Initial agreement is not a performance ceiling:** the best system beat the pre-consensus annotator agreement, and per-relation agreement correlated only weakly (0.15) with system accuracy. Revising guidelines after disagreements is worth doing, and raw early agreement should not be used to cap expectations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Roxana Girju, Preslav Nakov, Vivi Nastase, Stan Szpakowicz, Peter Turney and Deniz Yuret. 2007. SemEval-2007 Task 04: Classification of Semantic Relations between Nominals. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{girju2007semeval,
  title = {{SemEval-2007 Task 04: Classification of Semantic Relations between Nominals}},
  author = {Roxana Girju, Preslav Nakov, Vivi Nastase, Stan Szpakowicz, Peter Turney and Deniz Yuret},
  booktitle = {Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)},
  year = {2007},
  url = {https://aclanthology.org/S07-1003/}
}
```

- [← previous A Unified Local and Global Model for Discourse Coherence](https://md.research.elamur.ai/free/en/papers/elsner-2007-unified-local-global-model-discourse/)
- [next → Formalism-Independent Parser Evaluation with CCG and DepBank](https://md.research.elamur.ai/free/en/papers/clark-2007-formalism-independent-parser-evaluation-ccg/)
- [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/)

## References

15 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. T. Chklovski and P. Pantel. 2004. Verbocean: Mining the web for fine-grained semantic verb relations. In Proc. Conf. on Empirical Methods in Natural Language Processing, EMNLP-04, pages 33–40, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=T%2E+Chklovski+and+P%2E+Pantel%2E+2004%2E+Verbocean%3A+Mining+the+web+for+fine%2Dgrained+semantic+verb+relations%2E+In+Proc%2E+Conf%2E+on+Empirical+Methods+in+Natural+Language+Processing%2C+EMNLP%2D04%2C+pages+33%E2%80%9340%2C+Barcelo)
2. R. Girju, D. Moldovan, M. Tatu, and D. Antohe. 2005. On the semantics of noun compounds. Computer Speech and Language, 19:479–496. [find](https://scholar.google.com/scholar?q=R%2E+Girju%2C+D%2E+Moldovan%2C+M%2E+Tatu%2C+and+D%2E+Antohe%2E+2005%2E+On+the+semantics+of+noun+compounds%2E+Computer+Speech+and+Language%2C+19%3A479%E2%80%93496%2E)
3. M. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proc. 14th International Conf. on Computational Linguistics (COLING-92), pages 539–545. [find](https://scholar.google.com/scholar?q=M%2E+Hearst%2E+1992%2E+Automatic+acquisition+of+hyponyms+from+large+text+corpora%2E+In+Proc%2E+14th+International+Conf%2E+on+Computational+Linguistics+%28COLING%2D92%29%2C+pages+539%E2%80%93545%2E)
4. M. Lapata. 2002. The disambiguation of nominalizations. Computational Linguistics, 28(3):357–388. [find](https://scholar.google.com/scholar?q=M%2E+Lapata%2E+2002%2E+The+disambiguation+of+nominalizations%2E+Computational+Linguistics%2C+28%283%29%3A357%E2%80%93388%2E)
5. D.D. Lewis. 1991. Evaluating text categorization. In Proceedings of the Speech and Natural Language Workshop, pages 312–318, Asilomar. [find](https://scholar.google.com/scholar?q=D%2ED%2E+Lewis%2E+1991%2E+Evaluating+text+categorization%2E+In+Proceedings+of+the+Speech+and+Natural+Language+Workshop%2C+pages+312%E2%80%93318%2C+Asilomar%2E)
6. D. Moldovan, A. Badulescu, M. Tatu, D. Antohe, and R. Girju. 2004. Models for the semantic classification of noun phrases. In Proc. Computational Lexical Semantics Workshop at HLT-NAACL 2004, pages 60–67, Boston, MA. [find](https://scholar.google.com/scholar?q=D%2E+Moldovan%2C+A%2E+Badulescu%2C+M%2E+Tatu%2C+D%2E+Antohe%2C+and+R%2E+Girju%2E+2004%2E+Models+for+the+semantic+classification+of+noun+phrases%2E+In+Proc%2E+Computational+Lexical+Semantics+Workshop+at+HLT%2DNAACL+2004%2C+pages+60)
7. P. Nakov and M. Hearst. 2006. Using verbs to characterize noun-noun relations. In Proc. Twelfth International Conf. in Artificial Intelligence (AIMSA-06), pages 233–244, Varna,Bulgaria. [find](https://scholar.google.com/scholar?q=P%2E+Nakov+and+M%2E+Hearst%2E+2006%2E+Using+verbs+to+characterize+noun%2Dnoun+relations%2E+In+Proc%2E+Twelfth+International+Conf%2E+in+Artificial+Intelligence+%28AIMSA%2D06%29%2C+pages+233%E2%80%93244%2C+Varna%2CBulgaria%2E)
8. V. Nastase and S. Szpakowicz. 2003. Exploring noun-modifier semantic relations. In Fifth International Workshop on Computational Semantics (IWCS-5), pages 285–301, Tilburg, The Netherlands. [find](https://scholar.google.com/scholar?q=V%2E+Nastase+and+S%2E+Szpakowicz%2E+2003%2E+Exploring+noun%2Dmodifier+semantic+relations%2E+In+Fifth+International+Workshop+on+Computational+Semantics+%28IWCS%2D5%29%2C+pages+285%E2%80%93301%2C+Tilburg%2C+The+Netherlands%2E)
9. V. Nastase, J. Sayyad-Shirabad, M. Sokolova, and S. Szpakowicz. 2006. Learning noun-modifier semantic relations with corpus-based and WordNet-based features. In Proc. 21st National Conf. on Artificial Intelligence (AAAI 2006), pages 781–787, Boston, MA. [find](https://scholar.google.com/scholar?q=V%2E+Nastase%2C+J%2E+Sayyad%2DShirabad%2C+M%2E+Sokolova%2C+and+S%2E+Szpakowicz%2E+2006%2E+Learning+noun%2Dmodifier+semantic+relations+with+corpus%2Dbased+and+WordNet%2Dbased+features%2E+In+Proc%2E+21st+National+Conf%2E+on+Artificial)
10. B. Rosario and M. Hearst. 2001. Classifying the semantic relations in noun-compounds via domain-specific lexical hierarchy. In Proc. 2001 Conf. on Empirical Methods in Natural Language Processing (EMNLP-01), pages 82–90. [find](https://scholar.google.com/scholar?q=B%2E+Rosario+and+M%2E+Hearst%2E+2001%2E+Classifying+the+semantic+relations+in+noun%2Dcompounds+via+domain%2Dspecific+lexical+hierarchy%2E+In+Proc%2E+2001+Conf%2E+on+Empirical+Methods+in+Natural+Language+Processing+%28EMN)
11. B. Rosario, M. Hearst, and C. Fillmore. 2002. The descent of hierarchy, and selection in relational semantics. In Proc. 40th Annual Meeting of the Association for Computational Linguistics (ACL-02), pages 417– 424, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=B%2E+Rosario%2C+M%2E+Hearst%2C+and+C%2E+Fillmore%2E+2002%2E+The+descent+of+hierarchy%2C+and+selection+in+relational+semantics%2E+In+Proc%2E+40th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%2D02%29%2C+p)
12. M. Stephens, M. Palakal, S. Mukhopadhyay, and R. Raje. 2001. Detecting gene relations from MEDLINE abstracts. In Proc. Sixth Annual Pacific Symposium on Biocomputing, pages 483–496. [find](https://scholar.google.com/scholar?q=M%2E+Stephens%2C+M%2E+Palakal%2C+S%2E+Mukhopadhyay%2C+and+R%2E+Raje%2E+2001%2E+Detecting+gene+relations+from+MEDLINE+abstracts%2E+In+Proc%2E+Sixth+Annual+Pacific+Symposium+on+Biocomputing%2C+pages+483%E2%80%93496%2E)
13. M. Tatu and D. Moldovan. 2005. A semantic approach to recognizing textual entailment. In Proc. Human Language Technology Conf. and Conf. on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005), pages 371–378, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=M%2E+Tatu+and+D%2E+Moldovan%2E+2005%2E+A+semantic+approach+to+recognizing+textual+entailment%2E+In+Proc%2E+Human+Language+Technology+Conf%2E+and+Conf%2E+on+Empirical+Methods+in+Natural+Language+Processing+%28HLT%2FEMNLP+)
14. P.D. Turney and M.L. Littman. 2005. Corpus-based learning of analogies and semantic relations. Machine Learning, 60(1-3):251–278. [find](https://scholar.google.com/scholar?q=P%2ED%2E+Turney+and+M%2EL%2E+Littman%2E+2005%2E+Corpus%2Dbased+learning+of+analogies+and+semantic+relations%2E+Machine+Learning%2C+60%281%2D3%29%3A251%E2%80%93278%2E)
15. P.D. Turney. 2005. Measuring semantic similarity by latent relational analysis. In Proc. Nineteenth International Joint Conf. on Artificial Intelligence (IJCAI-05), pages 1136–1141, Edinburgh, Scotland. [find](https://scholar.google.com/scholar?q=P%2ED%2E+Turney%2E+2005%2E+Measuring+semantic+similarity+by+latent+relational+analysis%2E+In+Proc%2E+Nineteenth+International+Joint+Conf%2E+on+Artificial+Intelligence+%28IJCAI%2D05%29%2C+pages+1136%E2%80%931141%2C+Edinburgh%2C+Scotlan)

*Updated 24 October 2026*