--- title: "SemEval-2010 Task 8 — Hendrickx 2010" description: "This paper presents a shared evaluation for identifying which semantic relation links two marked nouns in a sentence." authors: ["Iris Hendrickx", "Su Nam Kim", "Zornitsa Kozareva", "Preslav Nakov", "Diarmuid Ó Séaghdha", "Sebastian Padó", "Marco Pennacchiotti", "Lorenza Romano", "Stan Szpakowicz"] year: 2010 venue: "Proceedings of the 5th International Workshop on Semantic Evaluation" source: "https://aclanthology.org/S10-1006/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals # SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals **Authors:** [Iris Hendrickx](https://md.research.elamur.ai/free/en/authors/iris-hendrickx/), [Su Nam Kim](https://md.research.elamur.ai/free/en/authors/su-nam-kim/), [Zornitsa Kozareva](https://md.research.elamur.ai/free/en/authors/zornitsa-kozareva/), [Preslav Nakov](https://md.research.elamur.ai/free/en/authors/preslav-nakov/), Diarmuid Ó Séaghdha, [Sebastian Padó](https://md.research.elamur.ai/free/en/authors/sebastian-pado/), [Marco Pennacchiotti](https://md.research.elamur.ai/free/en/authors/marco-pennacchiotti/), [Lorenza Romano](https://md.research.elamur.ai/free/en/authors/lorenza-romano/), Stan Szpakowicz · **Year:** 2010 · **Venue:** Proceedings of the 5th International Workshop on Semantic Evaluation · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/S10-1006/) · [PDF](https://aclanthology.org/S10-1006.pdf) **Citations:** [283](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary This paper presents a shared evaluation for identifying which semantic relation links two marked nouns in a sentence. For example, in "tea from dried ginseng", the relation is Entity-Origin. It extends SemEval-1 Task 4, which used one binary dataset per relation. Here, systems must choose among nine relations plus OTHER and also determine argument direction, without being told which noun fills which slot. The organizers built the data in three rounds. They harvested candidate sentences from the Web with many search patterns. Two people then annotated each candidate independently, and disagreements were adjudicated or the example was dropped. The result is a corpus of 10,717 examples. Ten teams submitted 28 runs, scored by macro-averaged F1 with directionality. The UTD system won with just over 82%, well ahead of the next entries. Performance rose with more training data, and richer semantic resources tended to help modestly. The dataset matters because it offers a common, larger benchmark for relation classification, which supports applications such as information extraction and textual entailment. ## Key points - **Relation inventory:** Cause-Effect, Instrument-Agency, Product-Producer, Content-Container, Entity-Origin, Entity-Destination, Component-Whole, Member-Collection and Message-Topic, plus OTHER. Arguments are common-noun heads in local syntactic contexts. Negated and modal cases are excluded. - **Data split:** 8,000 training and 2,717 test instances. The training set includes 891 re-annotated items from SemEval-1. Learning curves were reported for TD1–TD4 (1,000 / 2,000 / 4,000 / 8,000 examples). - **Annotation agreement:** inter-annotator agreement (exact match) ranged from about 58% (Entity-Origin) to 95.8% (Content-Container). - **Results:** UTD scored above 82%, FBK IRST 77.62% and ISI 77.57%. A random baseline reached about 6%, and a Naive Bayes baseline beat two systems. A majority vote of the top three systems gave 82.79%. - **Per-relation and architecture findings:** Cause-Effect was easiest, while Instrument-Agency and Product-Producer were hardest. Most systems used MaxEnt or SVM classifiers with WordNet, Google n-grams or Cyc. Building the task took over 1,000 person-hours. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 283 papers of our library cite it (of the 140884 whose bibliographies we read); 21 of them have a page here, 262 not yet. - [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://md.research.elamur.ai/free/en/papers/alt-2020-tacred-revisited-thorough-evaluation-tacred/) - [Attention Guided Graph Convolutional Networks for Relation Extraction](https://md.research.elamur.ai/free/en/papers/guo-2019-attention-guided-graph-convolutional-networks/) - [What Happens To BERT Embeddings During Fine-tuning?](https://md.research.elamur.ai/free/en/papers/merchant-2020-what-happens-bert-embeddings-during/) - [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://md.research.elamur.ai/free/en/papers/miwa-2016-end-end-relation-extraction-lstms/) - [PATTY: A Taxonomy of Relational Patterns with Semantic Types](https://md.research.elamur.ai/free/en/papers/nakashole-2012-patty-taxonomy-relational-patterns-semantic/) - [Reasoning with Latent Structure Refinement for Document-Level Relation Extraction](https://md.research.elamur.ai/free/en/papers/nan-2020-reasoning-latent-structure-refinement-document/) - [Relation Extraction: Perspective from Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2015-relation-extraction-perspective-convolutional-neural/) - [RelEx-PT: A Portuguese Sentence-Level Relation Extraction Dataset](https://md.research.elamur.ai/free/en/papers/pinto-2026-relex-pt-portuguese-sentence-level/) - [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) - [Classifying Relations by Ranking with Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/santos-2015-classifying-relations-ranking-convolutional-neural/) - [Improving Hypernymy Detection with an Integrated Path-based and Distributional Method](https://md.research.elamur.ai/free/en/papers/shwartz-2016-improving-hypernymy-detection-integrated-path/) - [Semantic Compositionality through Recursive Matrix-Vector Spaces](https://md.research.elamur.ai/free/en/papers/socher-2012-semantic-compositionality-through-recursive-matrix/) - [BERT Rediscovers the Classical NLP Pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/) - [What do you learn from context? Probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/) - [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://md.research.elamur.ai/free/en/papers/wan-2023-gpt-re-context-learning-relation/) - [Relation Classification via Multi-Level Attention CNNs](https://md.research.elamur.ai/free/en/papers/wang-2016-relation-classification-multi-level-attention/) - [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://md.research.elamur.ai/free/en/papers/wang-2023-instructuie-multi-task-instruction-tuning/) - [Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths](https://md.research.elamur.ai/free/en/papers/xu-2015-classifying-relations-long-short-term/) - [Relation Classification via Convolutional Deep Neural Network](https://md.research.elamur.ai/free/en/papers/zeng-2014-relation-classification-convolutional-deep-neural/) - [Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism](https://md.research.elamur.ai/free/en/papers/zeng-2018-extracting-relational-facts-end-end/) - [Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification](https://md.research.elamur.ai/free/en/papers/zhou-2016-attention-based-bidirectional-long-short/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{hendrickx2010semeval, title = {{SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals}}, author = {Hendrickx, Iris and Kim, Su Nam and Kozareva, Zornitsa and Nakov, Preslav and Séaghdha, Diarmuid Ó and Padó, Sebastian and Pennacchiotti, Marco and Romano, Lorenza and Szpakowicz, Stan}, booktitle = {Proceedings of the 5th International Workshop on Semantic Evaluation}, year = {2010}, url = {https://aclanthology.org/S10-1006/} } ``` - [← previous SemEval-2010 Task 5 : Automatic Keyphrase Extraction from Scientific Articles](https://md.research.elamur.ai/free/en/papers/kim-2010-semeval-2010-task-5-automatic/) - [next → Semi-Supervised Recognition of Sarcasm in Twitter and Amazon](https://md.research.elamur.ai/free/en/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/) - [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/) ## References 5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. X. Carreras and L. Màrquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proc. CoNLL-04, Boston, MA. [find](https://scholar.google.com/scholar?q=X%2E+Carreras+and+L%2E+Ma%CC%80rquez%2E+2004%2E+Introduction+to+the+CoNLL%2D2004+shared+task%3A+Semantic+role+labeling%2E+In+Proc%2E+CoNLL%2D04%2C+Boston%2C+MA%2E) 2. I. Dagan, B. Dolan, B. Magnini, and D. Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 15(4):i–xvii. [find](https://scholar.google.com/scholar?q=I%2E+Dagan%2C+B%2E+Dolan%2C+B%2E+Magnini%2C+and+D%2E+Roth%2E+2009%2E+Recognizing+textual+entailment%3A+Rational%2C+evaluation+and+approaches%2E+Natural+Language+Engineering%2C+15%284%29%3Ai%E2%80%93xvii%2E) 3. R. Girju, P. Nakov, V. Nastase, S. Szpakowicz, P. Turney, and D. Yuret. 2009. Classification of semantic relations between nominals. Language Resources and Evaluation, 43(2):105–121. [find](https://scholar.google.com/scholar?q=R%2E+Girju%2C+P%2E+Nakov%2C+V%2E+Nastase%2C+S%2E+Szpakowicz%2C+P%2E+Turney%2C+and+D%2E+Yuret%2E+2009%2E+Classification+of+semantic+relations+between+nominals%2E+Language+Resources+and+Evaluation%2C+43%282%29%3A105%E2%80%93121%2E) 4. I. Hendrickx, S. Kim, Z. Kozareva, P. Nakov, D. Ó Séaghdha, S. Padó, M. Pennacchiotti, L. Romano, and S. Szpakowicz. 2009. SemEval-2010 Task 8: Multi-way classification of semantic relations between pairs of nominals. In Proc. NAACL Workshop on Semantic Evaluations, Boulder, CO. [find](https://scholar.google.com/scholar?q=I%2E+Hendrickx%2C+S%2E+Kim%2C+Z%2E+Kozareva%2C+P%2E+Nakov%2C+D%2E+O%CC%81+Se%CC%81aghdha%2C+S%2E+Pado%CC%81%2C+M%2E+Pennacchiotti%2C+L%2E+Romano%2C+and+S%2E+Szpakowicz%2E+2009%2E+SemEval%2D2010+Task+8%3A+Multi%2Dway+classification+of+semantic+relations+betwee) 5. J. Kim, T. Ohta, S. Pyysalo, Y. Kano, and J. Tsujii. 2009. Overview of BioNLP’09 shared task on event extraction. In Proc. BioNLP-09, Boulder, CO. [find](https://scholar.google.com/scholar?q=J%2E+Kim%2C+T%2E+Ohta%2C+S%2E+Pyysalo%2C+Y%2E+Kano%2C+and+J%2E+Tsujii%2E+2009%2E+Overview+of+BioNLP%E2%80%9909+shared+task+on+event+extraction%2E+In+Proc%2E+BioNLP%2D09%2C+Boulder%2C+CO%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2009-overview-bionlp-09-shared-task/)