--- title: "The Senseval-3 English lexical sample task — Mihalcea 2004" description: "This paper reports on the English lexical sample task at SENSEVAL-3, a shared evaluation of targeted Word Sense Disambiguation (WSD)." authors: ["Rada Mihalcea", "Timothy Chklovski", "Adam Kilgarriff"] year: 2004 venue: "Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text" source: "https://aclanthology.org/W04-0807/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mihalcea-2004-senseval-3-english-lexical-sample/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mihalcea-2004-senseval-3-english-lexical-sample/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2004](https://md.research.elamur.ai/free/en/papers/2004/) / The Senseval-3 English lexical sample task # The Senseval-3 English lexical sample task **Authors:** [Rada Mihalcea](https://md.research.elamur.ai/free/en/authors/rada-mihalcea/), Timothy Chklovski, [Adam Kilgarriff](https://md.research.elamur.ai/free/en/authors/adam-kilgarriff/) · **Year:** 2004 · **Venue:** Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W04-0807/) · [PDF](https://aclanthology.org/W04-0807.pdf) · [OpenAlex](https://openalex.org/W1525367170) **Citations:** 221 citations (OpenAlex, 2026-10-14) · [90](https://md.research.elamur.ai/free/en/papers/mihalcea-2004-senseval-3-english-lexical-sample/cited-by/) in our library, of the 140884 papers whose bibliographies we read ## Summary This paper reports on the English lexical sample task at SENSEVAL-3, a shared evaluation of targeted Word Sense Disambiguation (WSD). Systems must assign the correct sense to selected ambiguous words in context. The organizers made two main changes from earlier rounds. First, annotations came from Web volunteers through the Open Mind Word Expert (OMWE) system rather than from trained lexicographers. Second, verbs used the Wordsmyth dictionary instead of WordNet. Example sentences were drawn from the BNC, and multi-word expressions were filtered out so that only single-word targets remained. Volunteer agreement was close to figures from earlier expert-built data, which suggests crowd annotation is a workable way to build such corpora. The task attracted 27 teams and 47 submissions. Most systems beat the most-frequent-sense baseline, including several unsupervised ones, and top results often came from ensembles of classifiers. The data were released publicly. This makes the task a reference benchmark and a demonstration of low-cost sense annotation. ## Key points - The sample covers 57 words: 20 nouns, 32 verbs and 5 adjectives. They average 6.47 fine-grained and 4.96 coarse-grained senses. Nouns and adjectives use WordNet 1.7.1, and verbs use Wordsmyth. - Annotation followed a "tag until two agree" rule, with at most four annotations per item. Contributors could also choose "unclear" or "none of the above." - Agreement between the first two taggings was 62.8% on single words and about 67.3% overall. SENSEVAL-2 reported 66.5%. With a chance agreement of 0.20, micro-kappa was 0.58 and macro-kappa was 0.35. - The most-frequent-sense baseline scored 55.2% fine-grained and 64.5% coarse-grained. The best reported system reached 72.9% fine-grained and 79.3% coarse-grained. - Many leading systems combined several learners, such as SVM, Naive Bayes, maximum entropy, decision lists and kernel methods. The strongest unsupervised entry, wsdiit, reached 54.7% fine-grained. ## Interesting ideas - **Hide other annotators' answers.** Contributors never saw how others had labeled an item, which avoids herd bias. This is a cheap safeguard for any crowdsourced labeling effort, such as Ukrainian NER or sentiment data. - **Adaptive redundancy ("tag until two agree", capped at four).** Extra labels are collected only when the first annotators disagree. This keeps costs low on easy items and concentrates effort on ambiguous ones. - **Swap the sense inventory to test what makes a task hard.** The organizers changed only the verb dictionary to probe whether poor verb scores came from WordNet's very fine sense distinctions. Varying the label scheme while holding the task fixed is a useful diagnostic for any classification benchmark. - **Separate subproblems before evaluation.** Removing multi-word expressions meant that sense tagging was measured on its own, without being mixed with collocation detection. Cleanly isolating components makes system scores easier to interpret. - **Report both micro and macro kappa, plus fine and coarse scoring.** Showing agreement and accuracy at several aggregation levels and granularities reveals how much a result depends on how it is measured. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mihalcea2004senseval, title = {{The Senseval-3 English lexical sample task}}, author = {Mihalcea, Rada and Chklovski, Timothy and Kilgarriff, Adam}, booktitle = {Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text}, year = {2004}, url = {https://aclanthology.org/W04-0807/} } ``` - [← previous Deterministic Dependency Parsing of English Text](https://md.research.elamur.ai/free/en/papers/aclbulk-2004-nivre-deterministic-dependency-parsing-english-text/) - [next → A Mention-Synchronous Coreference Resolution Algorithm Based On the Bell Tree](https://md.research.elamur.ai/free/en/papers/luo-2004-mention-synchronous-coreference-resolution-algorithm/) - [↑ back to papers of 2004](https://md.research.elamur.ai/free/en/papers/2004/) ## References 5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. J. Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249– 254. [find](https://scholar.google.com/scholar?q=J%2E+Carletta%2E+1996%2E+Assessing+agreement+on+classification+tasks%3A+The+kappa+statistic%2E+Computational+Linguistics%2C+22%282%29%3A249%E2%80%93+254%2E) [our card](https://md.research.elamur.ai/free/en/papers/carletta-1996-assessing-agreement-classification-tasks-kappa/) 2. T. Chklovski and R. Mihalcea. 2002. Building a sense tagged corpus with Open Mind Word Expert. In Proceedings of the ACL 2002 Workshop on ”Word Sense Disambiguation: Recent Successes and Future Directions”, Philadelphia, July. [find](https://scholar.google.com/scholar?q=T%2E+Chklovski+and+R%2E+Mihalcea%2E+2002%2E+Building+a+sense+tagged+corpus+with+Open+Mind+Word+Expert%2E+In+Proceedings+of+the+ACL+2002+Workshop+on+%E2%80%9DWord+Sense+Disambiguation%3A+Recent+Successes+and+Future+Direct) 3. A. Kilgarriff and M. Palmer, editors. 2000. Computer and the Humanities. Special issue: SENSEVAL. Evaluating Word Sense Disambiguation programs, volume 34, April. [find](https://scholar.google.com/scholar?q=A%2E+Kilgarriff+and+M%2E+Palmer%2C+editors%2E+2000%2E+Computer+and+the+Humanities%2E+Special+issue%3A+SENSEVAL%2E+Evaluating+Word+Sense+Disambiguation+programs%2C+volume+34%2C+April%2E) 4. G. Miller. 1995. Wordnet: A lexical database. Communication of the ACM, 38(11):39–41. [find](https://scholar.google.com/scholar?q=G%2E+Miller%2E+1995%2E+Wordnet%3A+A+lexical+database%2E+Communication+of+the+ACM%2C+38%2811%29%3A39%E2%80%9341%2E) 5. J. Preiss and D. Yarowsky, editors. 2001. Proceedings of SENSEVAL-2, Association for Computational Linguistics Workshop, Toulouse, France. [find](https://scholar.google.com/scholar?q=J%2E+Preiss+and+D%2E+Yarowsky%2C+editors%2E+2001%2E+Proceedings+of+SENSEVAL%2D2%2C+Association+for+Computational+Linguistics+Workshop%2C+Toulouse%2C+France%2E)