--- title: "Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition — Derczynski 2017" description: "The paper examines a known weakness of named entity recognition: systems score well on standard benchmarks mainly… 2 references, cited by 173 papers here." authors: ["Leon Derczynski", "Eric Nichols", "Marieke van Erp", "Nut Limsopatham"] year: 2017 venue: "Proceedings of the 3rd Workshop on Noisy User-generated Text" source: "https://aclanthology.org/W17-4418/" doi: "10.18653/v1/w17-4418" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/derczynski-2017-results-wnut2017-shared-task-novel/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/derczynski-2017-results-wnut2017-shared-task-novel/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition # Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition **Authors:** [Leon Derczynski](https://md.research.elamur.ai/free/en/authors/leon-derczynski/), [Eric Nichols](https://md.research.elamur.ai/free/en/authors/eric-nichols/), Marieke van Erp and Nut Limsopatham · **Year:** 2017 · **Venue:** Proceedings of the 3rd Workshop on Noisy User-generated Text · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W17-4418/) · [PDF](https://aclanthology.org/W17-4418.pdf) · [DOI 10.18653/v1/w17-4418](https://doi.org/10.18653/v1/w17-4418) · [OpenAlex](https://openalex.org/W2760505947) **Citations:** 361 citations (OpenAlex, 2026-10-14) · [173](https://md.research.elamur.ai/free/en/papers/derczynski-2017-results-wnut2017-shared-task-novel/cited-by/) in our library, of the 140,884 papers whose bibliographies we read ## Summary The paper examines a known weakness of named entity recognition: systems score well on standard benchmarks mainly because those benchmarks repeat frequent, familiar names, and they break down on rare or newly appearing entities in noisy social-media text. The WNUT 2017 shared task set out to measure how well systems generalise to such unseen names. The organisers trained on an existing Twitter NER corpus. They built new development and test sets from YouTube, Twitter, Reddit and StackExchange comments, and removed every text containing a surface form that was already labelled as an entity in the training set. Crowd workers annotated six entity types. Systems were scored with standard entity-level F1 and with a new surface-form F1 that counts each distinct (name, type) pair only once. Seven teams took part. The best result, from UH-RiTUAL, was roughly 42 entity F1, far below typical newswire numbers. This confirms that emerging and long-tail entities remain hard. The work matters because it provides a reusable benchmark and metric that reward generalisation rather than memorisation. ## Key points - **Training data:** the Ritter et al. (2011) Twitter corpus, with 1,000 tweets and 65,124 tokens. - **Development set:** YouTube comments only, with 1,008 documents, 15,734 tokens and 835 entities. - **Test set:** Twitter, Reddit and StackExchange, with 1,287 documents, 23,394 tokens and 1,040 entities. - **Entity classes:** person, location (covering GPE and facility), corporation, product, creative-work and group. - **Annotation:** done through CrowdFlower using the GATE crowdsourcing plugin, with three annotators assigned to each document. - **Results:** seven systems were scored on F1 (entity) and F1 (surface). The top system was UH-RiTUAL with 41.86 / 40.24, followed by SpinningBytes with 40.78 / 39.33. The weakest was Drexel-CCI with 26.30 / 25.26. - **Error analysis:** - Systems often cut off names that contain common words, such as tagging only "Andrew" in "Andrew Little". - They mislabelled non-English names, for example the Kashmiri village "Mahazgund". - Corporation and creative-work were the hardest classes. Typical problems were confusion between corporation, group and product, and long titles built from ordinary words. ## Interesting ideas - **Novelty by construction:** collect all entity strings from the training set into a blacklist and drop any evaluation text that contains one of them. This cheaply guarantees a truly unseen test set. It could be applied to any Ukrainian NER or gazetteer-heavy task where memorisation inflates scores. - **Surface-form F1:** score unique (surface form, type) tuples rather than individual mentions, so a frequent name such as a capital city counts only once. This exposes whether a model handles diverse names or just the head of the distribution. It is a useful extra metric wherever entity frequencies follow a Zipfian distribution. - **Deliberate domain drift between splits:** use different source mixtures for development and test data, such as YouTube for development and Reddit, Twitter and StackExchange for test. This creates an intentionally hostile setting that tests robustness instead of overfitting to one platform. It is a good design for stress-testing models before real-world deployment. - **Pre-filtering for entity-bearing text:** keep only sentences in which two independent POS taggers both mark at least one proper noun (NNP). This raises the density of entities before paying for annotation, and the same trick could cut crowdsourcing costs for new corpora. - **Sampling around specific events and regions:** draw data from periods of recent disasters and from city- or country-level forums to surface novel place names. This is worth borrowing for geography-oriented datasets that need rare local toponyms. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [🏷️ Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) ## Institutions - [University of Sheffield](https://md.research.elamur.ai/free/en/universities/university-of-sheffield/) — University, Sheffield, United Kingdom · 67 works · 17,924 citations - [Vrije Universiteit Amsterdam](https://md.research.elamur.ai/free/en/universities/vrije-universiteit-amsterdam/) — University, Amsterdam, Netherlands · 31 works · 3,248 citations - [Honda](https://md.research.elamur.ai/free/en/companies/honda/) — Company, Tokyo, Japan · 2 works · 2,810 citations - [Honda Research Institute Japan Co., Ltd.](https://md.research.elamur.ai/free/en/companies/honda-research-institute-japan-co-ltd/) — Company, Wako, Japan · 2 works · 2,810 citations Also on the byline: [Accenture (Ireland)](https://ror.org/00hmyb194). ## Cite Cite as: Leon Derczynski, Eric Nichols, Marieke van Erp and Nut Limsopatham. 2017. Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition. Proceedings of the 3rd Workshop on Noisy User-generated Text. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{derczynski2017results, title = {{Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition}}, author = {Leon Derczynski, Eric Nichols, Marieke {van Erp} and Nut Limsopatham}, booktitle = {Proceedings of the 3rd Workshop on Noisy User-generated Text}, year = {2017}, doi = {10.18653/v1/w17-4418}, url = {https://doi.org/10.18653/v1/w17-4418} } ``` - [← previous Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/bryant-2017-automatic-annotation-evaluation-error-types/) - [next → Creating Training Corpora for NLG Micro-Planners](https://md.research.elamur.ai/free/en/papers/gardent-2017-creating-training-corpora-nlg-micro/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 2016. On Emerging Entity Detection. [find](https://scholar.google.com/scholar?q=2016%2E+On+Emerging+Entity+Detection%2E) 2. Gustavo Aguilar, Suraj Maharjan, Adrian Pastor López Monroy, and Thamar Solorio. 2017. A Multi-task Approach for Named Entity Recognition in Social Media Data. In Proceedings of the 3rd Workshop on Noisy, User-generated Text (W-NUT) at EMNLP. ACL. \#\# Conclusion [find](https://scholar.google.com/scholar?q=Gustavo+Aguilar%2C+Suraj+Maharjan%2C+Adrian+Pastor+Lo%CC%81pez+Monroy%2C+and+Thamar+Solorio%2E+2017%2E+A+Multi%2Dtask+Approach+for+Named+Entity+Recognition+in+Social+Media+Data%2E+In+Proceedings+of+the+3rd+Workshop+on+N)