--- title: "English Lexical Sample Task Description — Kilgarriff 2001" description: "The paper documents how the English lexical sample track of SENSEVAL-2 was built. 2 references, cited by 29 papers here." authors: ["Adam Kilgarriff"] year: 2001 venue: "Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems" source: "https://aclanthology.org/S01-1004/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kilgarriff-2001-english-lexical-sample-task-description/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kilgarriff-2001-english-lexical-sample-task-description/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2001](https://md.research.elamur.ai/free/en/papers/2001/) / English Lexical Sample Task Description # English Lexical Sample Task Description **Authors:** [Adam Kilgarriff](https://md.research.elamur.ai/free/en/authors/adam-kilgarriff/) · **Year:** 2001 · **Venue:** [Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems](https://md.research.elamur.ai/free/en/venues/senseval-2-international-workshop-on-evaluating-word-sense-disambiguation/) ([SENSEVAL-2 International Workshop on Evaluating Word Sense Disambiguation Systems 2001](https://md.research.elamur.ai/free/en/venues/senseval-2-international-workshop-on-evaluating-word-sense-disambiguation/2001/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/S01-1004/) · [PDF](https://aclanthology.org/S01-1004.pdf) · [OpenAlex](https://openalex.org/W1567165223) **Citations:** 78 citations (OpenAlex, 2026-10-14) · [29](https://md.research.elamur.ai/free/en/papers/kilgarriff-2001-english-lexical-sample-task-description/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper documents how the English lexical sample track of SENSEVAL-2 was built. It covered adjectives and nouns; verbs were handled with the all-words task. The organisers chose a fresh set of words spanning low, medium and high frequency. Contexts came mostly from the new edition of the British National Corpus, plus a small amount of Wall Street Journal text. The key change from SENSEVAL-1 was the sense inventory: the HECTOR lexicon was replaced by WordNet. Before use, a lexicographer revised the WordNet entries and merged senses that were too close to tell apart, and these edits were folded into WordNet 1.7. Recruited taggers, screened by a web-based test, annotated the data. Each item went to more annotators only when the first ones disagreed. Overall fine-grained inter-tagger agreement reached 85.5%. Even so, the best systems scored roughly 14% below their SENSEVAL-1 counterparts. The author attributes this drop to WordNet's less clear-cut sense distinctions. His conclusion is that WSD research should now focus on sense inventories rather than on algorithms. ## Key points - The sample had 15 adjectives and 29 nouns, giving 7,567 tagged instances (2,301 adjectives, 5,266 nouns), split 2:1 into training and test data in SENSEVAL XML format. - The target number of instances per word was 75 + 15n + 6m, where n is the number of senses and m the number of multiword expressions the word belongs to. - The annotation team had 3 professional lexicographers and 14 students or others. They were recruited through an online aptitude test whose gold answers came from SENSEVAL-1 data. - 66.5% of items needed only two taggings and 32.3% needed three; just 90 items needed four or five. Fine-grained ITA was 83.4% for adjectives and 86.3% for nouns. - Kappa was deliberately not reported. Results were published in two sets: the official submissions and later bug-fix submissions marked (R). ## Interesting ideas - Annotation effort scales with difficulty: start with two annotators and add one at a time until at least two agree on a tag, so effort concentrates on genuinely hard items. - Annotators worked with short human-readable mnemonics for each sense instead of bare sense numbers, which are easy to mix up. - Submitted answers were checked automatically against the permitted labels for that word, and invalid ones were sent back for correction. Taggers could also give several tags, or mark an item as Unassignable or Proper-name, rather than being forced into one choice. - The paper argues that the true ceiling for a task is replicability, meaning how well two independent teams using the same method agree, not agreement within a single team. - Sense inventories were cleaned by merging distinctions a lexicographer could not justify, on the view that disambiguation quality depends more on the inventory than on the algorithm. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Our silver-data pipeline could call an extra witness parser or a costlier LLM teacher tier only when the cheaper witnesses disagree, saving compute on easy sentences. - **I2** (world): WordNet-derived sense IDs in our ontology could carry curated mnemonic labels, making frame and entity explanations readable for people debugging the world model. - **I3** (learn): LLM teacher outputs could be validated against the allowed VerbNet or sense inventory and re-queried when out of range, with explicit "none fits" and "several fit" options kept in the label schema. - **I4** (learn): When estimating upper bounds for silver data, we could compare outputs from independently configured annotation pipelines instead of witnesses that share training data. - **I5** (world): Before building frames on WordNet senses, we could collapse poorly motivated sense splits so that downstream readers are not penalised for distinctions nobody can reliably make. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Brighton](https://md.research.elamur.ai/free/en/universities/university-of-brighton/) — University, Brighton, United Kingdom · 14 works · 2,860 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works ## Cite Cite as: Adam Kilgarriff. 2001. English Lexical Sample Task Description. Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{kilgarriff2001english, title = {{English Lexical Sample Task Description}}, author = {Adam Kilgarriff}, booktitle = {Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems}, year = {2001}, url = {https://aclanthology.org/S01-1004/} } ``` - [← previous A Corpus-Based Evaluation of Centering and Pronoun Resolution](https://md.research.elamur.ai/free/en/papers/tetreault-2001-corpus-based-evaluation-centering-pronoun/) - [next → SENSEVAL-2: Overview](https://md.research.elamur.ai/free/en/papers/edmonds-2001-senseval-2-overview/) - [↑ back to papers of 2001](https://md.research.elamur.ai/free/en/papers/2001/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Adam Kilgarriff and Joseph Rosenzweig. 2000. Framework and results for English SENSEVAL. Computers and the Humanities, 34(1-2):15-48. Special Issue on SENSEVAL, edited by Adam Kilgarriff and Martha Palmer. [find](https://scholar.google.com/scholar?q=Adam+Kilgarriff+and+Joseph+Rosenzweig%2E+2000%2E+Framework+and+results+for+English+SENSEVAL%2E+Computers+and+the+Humanities%2C+34%281%2D2%29%3A15%2D48%2E+Special+Issue+on+SENSEVAL%2C+edited+by+Adam+Kilgarriff+and+Martha+Pa) 2. Adam Kilgarriff. 1999. 95% replicability for manual word sense tagging. In Proc. EACL, pages 277-278, Bergen, June. [find](https://scholar.google.com/scholar?q=Adam+Kilgarriff%2E+1999%2E+95%25+replicability+for+manual+word+sense+tagging%2E+In+Proc%2E+EACL%2C+pages+277%2D278%2C+Bergen%2C+June%2E) *Updated 2 November 2026*