--- title: "SemEval-2007 Task-17 — Pradhan 2007" description: "This paper describes the organization and outcomes of SemEval-2007 Task-17, which jointly evaluated word sense disambiguation (WSD) and semantic role…" authors: ["Sameer Pradhan", "Edward Loper", "Dmitriy Dligach", "Martha Palmer"] year: 2007 venue: "Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)" source: "https://aclanthology.org/S07-1016/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/pradhan-2007-semeval-2007-task-17-english/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pradhan-2007-semeval-2007-task-17-english/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/) / SemEval-2007 Task-17: English Lexical Sample, SRL and All Words # SemEval-2007 Task-17: English Lexical Sample, SRL and All Words **Authors:** [Sameer Pradhan](https://md.research.elamur.ai/free/en/authors/sameer-pradhan/), Edward Loper, Dmitriy Dligach, [Martha Palmer](https://md.research.elamur.ai/free/en/authors/martha-palmer/) · **Year:** 2007 · **Venue:** Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/S07-1016/) · [PDF](https://aclanthology.org/S07-1016.pdf) · [OpenAlex](https://openalex.org/W2031026221) **Citations:** 297 citations (openalex-snapshot · 2026-10-14) · [145](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary This paper describes the organization and outcomes of SemEval-2007 Task-17, which jointly evaluated word sense disambiguation (WSD) and semantic role labeling (SRL) on English text from the Wall Street Journal. There were three subtasks. The fine-grained All-Words subtask used about 3,500 words from three Treebanked and PropBanked WSJ articles, which were newly sense-tagged with WordNet 2.1. The coarse-grained Lexical Sample subtask drew on OntoNotes sense annotations, which have high inter-annotator agreement. The SRL subtask covered a subset of the Lexical Sample verbs and labeled them with both PropBank and VerbNet roles. The results show a sharp contrast between the two WSD settings. Fine-grained All-Words systems scored only modestly above a most-frequent-sense baseline. Coarse-grained Lexical Sample systems came close to the human agreement ceiling. The organizers read this as support for grouping senses into coarser, more reliably annotated distinctions. Only two teams took part in the SRL subtask. Their results suggest that VerbNet roles, obtained through a PropBank-to-VerbNet mapping, can be predicted roughly as well as the original PropBank labels. ## Key points - **All-Words subtask:** - Data: 465 lemmas from three WSJ files, annotated with the STAMP tool. - Inter-annotator agreement: 72% on verbs and 86% on nouns. - Results: 14 systems took part. The top system was PNNL, a MaxEnt classifier, with F = 59.1. The most-frequent WordNet sense baseline scored 51.4. - **OntoNotes Lexical Sample subtask:** - Data: 100 lemmas (65 verbs, 35 nouns) with average agreement above 90%. - Splits: training data came from WSJ Sections 02-21 (22,281 instances) and test data from Sections 01, 22, 23 and 24 (4,851 instances). - Results: 13 systems took part. The best was NUS-ML (SVM) with F = 88.7, against a training-set most-frequent-sense baseline of 78.0. - Across lemmas, verbs scored about 10 points lower than nouns on average. - **SRL subtask:** - Setup: 50 verbs were annotated with PropBank arguments and with VerbNet roles generated via SemLink. Closed and Open participation tracks were offered. - Results: UBC-UPC reached roughly F 83.4–83.7 and RTV roughly 75.4–75.7. - All scores carry 95% bootstrap confidence intervals. Scoring used scorer2 for WSD and the CoNLL-2005 srl-eval.pl script for SRL. ## Interesting ideas - **Bootstrap confidence intervals on every score.** Each F-score is reported with an interval computed by bootstrap resampling. This makes it easy to see when neighboring systems are statistically indistinguishable, and it is cheap to adopt in any shared task or benchmark leaderboard. - **Removing a dominant item from the test set.** More than half of the SRL test instances involved the very easy predicate "say," so the organizers also reported scores with it excluded. This prevents one frequent, trivial item from inflating results, and the same check applies to any skewed dataset. - **Two label layers from one annotation via a mapping.** VerbNet roles were derived automatically from PropBank labels using SemLink, falling back to the PropBank label when no mapping existed. This yields a more generalizable label set without a second round of manual annotation, which is useful when building resources for a new language on top of existing ones. - **Splits aligned with standard parser training sections.** Training data was taken from the WSJ sections that off-the-shelf parsers are usually trained on, and test data from other sections. Participants could then use parser features without the test data leaking into those features. - **Separate closed and open tracks.** Distinguishing systems trained only on the provided data from systems allowed outside resources shows how much external data actually helps. In this task, the difference turned out to be negligible. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Word sense disambiguation](https://md.research.elamur.ai/free/en/topics/word-sense-disambiguation/) · [Semantic role labeling](https://md.research.elamur.ai/free/en/topics/semantic-role-labeling/) ## Cited by Cited by 145 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 137 not yet. - [Random Walks for Knowledge-Based Word Sense Disambiguation](https://md.research.elamur.ai/free/en/papers/agirre-2014-random-walks-knowledge-based-word/) - [Exemplification Modeling: Can You Give Me an Example, Please?](https://md.research.elamur.ai/free/en/papers/barba-2021-exemplification-modeling/) - [ChatGPT: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/) - [Generationary or «How We Went beyond Word Sense Inventories and Learned to Gloss»](https://md.research.elamur.ai/free/en/papers/lex-2020-bevilacqua-generationary/) - [DiBiMT: A Gold Evaluation Benchmark for Studying Lexical Ambiguity in Machine Translation](https://md.research.elamur.ai/free/en/papers/martelli-2025-dibimt-gold-evaluation-benchmark-studying/) - [Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison](https://md.research.elamur.ai/free/en/papers/onto-2017-raganato-word-sense-disambiguation-unified-evaluation/) - [Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/) - [MWE as WSD: Solving Multiword Expression Identification with Word Sense Disambiguation](https://md.research.elamur.ai/free/en/papers/tanner-2023-mwe-as-wsd-solving-multiword/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{pradhan2007semeval, title = {{SemEval-2007 Task-17: English Lexical Sample, SRL and All Words}}, author = {Pradhan, Sameer and Loper, Edward and Dligach, Dmitriy and Palmer, Martha}, booktitle = {Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007)}, year = {2007}, doi = {10.3115/1621474.1621490}, url = {https://doi.org/10.3115/1621474.1621490} } ``` - [← previous SemEval-2007 Task 19: Frame Semantic Structure Extraction](https://md.research.elamur.ai/free/en/papers/baker-2007-semeval-2007-task-19-frame/) - [next → Semi-automatic error analysis for large-scale statistical machine translation](https://md.research.elamur.ai/free/en/papers/kirchhoff-2007-semi-automatic-error-analysis-large/) - [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/) ## References 11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Xavier Carreras and Lluı́s M` arquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proceedings of CoNLL-2004. [find](https://scholar.google.com/scholar?q=Xavier+Carreras+and+Llu%C4%B1%CC%81s+M%60+arquez%2E+2004%2E+Introduction+to+the+CoNLL%2D2004+shared+task%3A+Semantic+role+labeling%2E+In+Proceedings+of+CoNLL%2D2004%2E) 2. Xavier Carreras and Lluı́s M` arquez. 2005. Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling. In Proceedings of CoNLL-2005. [find](https://scholar.google.com/scholar?q=Xavier+Carreras+and+Llu%C4%B1%CC%81s+M%60+arquez%2E+2005%2E+Introduction+to+the+CoNLL%2D2005+Shared+Task%3A+Semantic+Role+Labeling%2E+In+Proceedings+of+CoNLL%2D2005%2E) [our card](https://md.research.elamur.ai/free/en/papers/carreras-2005-introduction-conll-2005-shared-task/) 3. Jinying Chen, Andrew Schein, Lyle Ungar, and Martha Palmer. 2006. An empirical study of the behavior of active learning for word sense disambiguation. In Proceedings of HLT/NAACL. [find](https://scholar.google.com/scholar?q=Jinying+Chen%2C+Andrew+Schein%2C+Lyle+Ungar%2C+and+Martha+Palmer%2E+2006%2E+An+empirical+study+of+the+behavior+of+active+learning+for+word+sense+disambiguation%2E+In+Proceedings+of+HLT%2FNAACL%2E) 4. Timothy Chklovski and Rada Mihalcea. 2002. Building a sense tagged corpus with open mind word expert. In Proceedings of ACL-02 Workshop on WSD. [find](https://scholar.google.com/scholar?q=Timothy+Chklovski+and+Rada+Mihalcea%2E+2002%2E+Building+a+sense+tagged+corpus+with+open+mind+word+expert%2E+In+Proceedings+of+ACL%2D02+Workshop+on+WSD%2E) 5. Bart Decadt, Véronique Hoste, Walter Daelemans, and Antal Van den Bosch. 2004. GAMBL, genetic algorithm optimization of memory-based wsd. In Senseval-3. [find](https://scholar.google.com/scholar?q=Bart+Decadt%2C+Ve%CC%81ronique+Hoste%2C+Walter+Daelemans%2C+and+Antal+Van+den+Bosch%2E+2004%2E+GAMBL%2C+genetic+algorithm+optimization+of+memory%2Dbased+wsd%2E+In+Senseval%2D3%2E) 6. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of HLT/NAACL, June. [find](https://scholar.google.com/scholar?q=Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Lance+Ramshaw%2C+and+Ralph+Weischedel%2E+2006%2E+Ontonotes%3A+The+90%25+solution%2E+In+Proceedings+of+HLT%2FNAACL%2C+June%2E) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2009-ontonotes-90-solution/) 7. Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer. 2006. Extending VerbNet with novel verb classes. In LREC-06. [find](https://scholar.google.com/scholar?q=Karin+Kipper%2C+Anna+Korhonen%2C+Neville+Ryant%2C+and+Martha+Palmer%2E+2006%2E+Extending+VerbNet+with+novel+verb+classes%2E+In+LREC%2D06%2E) 8. Ken Litkowski. 2004. Senseval-3 task: Automatic labeling of semantic roles. In Proceedings of Senseval-3. [find](https://scholar.google.com/scholar?q=Ken+Litkowski%2E+2004%2E+Senseval%2D3+task%3A+Automatic+labeling+of+semantic+roles%2E+In+Proceedings+of+Senseval%2D3%2E) [our card](https://md.research.elamur.ai/free/en/papers/gildea-2002-automatic-labeling-semantic-roles/) 9. Edward Loper, Szu ting Yi, and Martha Palmer. 2007. Combining lexical resources: Mapping between propbank and verbnet. In Proceedings of the IWCS-7. [find](https://scholar.google.com/scholar?q=Edward+Loper%2C+Szu+ting+Yi%2C+and+Martha+Palmer%2E+2007%2E+Combining+lexical+resources%3A+Mapping+between+propbank+and+verbnet%2E+In+Proceedings+of+the+IWCS%2D7%2E) 10. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: A corpus annotated with semantic roles. Computational Linguistics, 31(1):71–106. [find](https://scholar.google.com/scholar?q=Martha+Palmer%2C+Daniel+Gildea%2C+and+Paul+Kingsbury%2E+2005%2E+The+proposition+bank%3A+A+corpus+annotated+with+semantic+roles%2E+Computational+Linguistics%2C+31%281%29%3A71%E2%80%93106%2E) 11. Martha Palmer. 2000. Consistent criteria for sense distinctions. Computers and the Humanities, 34(1-1):217–222. [find](https://scholar.google.com/scholar?q=Martha+Palmer%2E+2000%2E+Consistent+criteria+for+sense+distinctions%2E+Computers+and+the+Humanities%2C+34%281%2D1%29%3A217%E2%80%93222%2E)