---
title: "Extending English ACE 2005 Corpus Annotation with"
description: "Coreference systems increasingly use Wikipedia as a source of semantic knowledge, such as… 13 references, cited by 5 papers here."
authors: ["Luisa Bentivogli", "Pamela Forner", "Claudio Giuliano", "Alessandro Marchetti", "Emanuele Pianta", "Kateryna Tymoshenko"]
year: 2010
venue: "Proceedings of the 2nd Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources"
source: "https://aclanthology.org/W10-3503/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bentivogli-2010-extending-english-ace-2005-corpus/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bentivogli-2010-extending-english-ace-2005-corpus/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) / Extending English ACE 2005 Corpus Annotation with Ground-truth Links to Wikipedia

# Extending English ACE 2005 Corpus Annotation with Ground-truth Links to Wikipedia

**Authors:** [Luisa Bentivogli](https://md.research.elamur.ai/free/en/authors/luisa-bentivogli/), Pamela Forner, [Claudio Giuliano](https://md.research.elamur.ai/free/en/authors/claudio-giuliano/), Alessandro Marchetti, Emanuele Pianta and [Kateryna Tymoshenko](https://md.research.elamur.ai/free/en/authors/kateryna-tymoshenko/) · **Year:** 2010 · **Venue:** [Proceedings of the 2nd Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2010](https://md.research.elamur.ai/free/en/venues/coling/2010/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W10-3503/) · [PDF](https://aclanthology.org/W10-3503.pdf) · [OpenAlex](https://openalex.org/W189596627)

**Citations:** 32 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/bentivogli-2010-extending-english-ace-2005-corpus/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Coreference systems increasingly use Wikipedia as a source of semantic knowledge, such as categories and anchor-text aliases. To use it, a system must first link each mention to the right page. Measuring how accurately this linking works inside a coreference pipeline requires a corpus whose mentions carry gold Wikipedia links. Existing wikification test sets were either partial, too small, or annotated far beyond entity mentions.

The authors therefore add a manual layer to the English ACE 2005 corpus. All named (NAM) and nominal (NOM) mentions, about 29,300 in total, receive links to Wikipedia pages. Names point to the page for the specific referent. Nominals point to the page for the word's sense in context, given as a ranked list from most specific to more general. The work was roughly 55% complete when the paper was written.

Two annotators disagreed on only about 8% of cases. Most disagreements came from different readings of the context rather than from senses missing in Wikipedia. The authors take this as quantitative evidence that Wikipedia covers newswire mentions well. The layer reuses an established coreference benchmark at modest extra cost.

## Key points
- English ACE 2005 has 599 documents from news, newswire, web and transcribed speech. It contains 15,382 entities and 43,624 mentions (person, location, organization) with in-document coreference IDs.
- Only NAM and NOM mentions are linked; pronouns are excluded. Only the syntactic head is tagged, but multiword units like "flight attendant" are linked as a whole (Flight_attendant).
- A NAM link is made once per document, on the first occurrence. It is then copied to coreferent names using the existing ACE coreference IDs.
- Disambiguation pages may not be used as links. Annotators use a "Missing sense" tag when no fitting page exists.
- Disagreements are classified into three types:
  - no common link;
  - different first link;
  - same first link but extra links added.
- About 61.5% of disagreements stem from differing interpretations of context, not from gaps in Wikipedia.

## Interesting ideas
- Link a nominal to its concept, not its referent: "the capital" meaning Baghdad gets Capital_(political), keeping sense and reference as separate layers.
- Store a ranked list of senses (most specific first, then more general) instead of one forced choice, so partial matches can still be scored.
- Annotate one mention per coreference chain and propagate the label through existing chains, which cuts annotation effort a lot.
- Make "missing sense" an explicit label rather than forcing a near-miss choice, which keeps inventory gaps visible.
- Split annotator disagreements by type (no match / different first choice / extra links) and by cause (inventory gap vs. context ambiguity).

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): Store a nominal's ontology class separately from its coreference link to a world entity, so each can be explained and checked independently.
- **I2** (world): Let WordNet/Wikidata grounding emit a ranked specific-to-general chain (President_of_the_United_States → President), so later reasoning can fall back to the broader class.
- **I3** (learn): When building silver data, label one representative mention per agreed chain and copy the label to the rest, saving LLM-teacher calls.
- **I4** (world): Add an explicit "no ontology node" outcome so gaps in the compiled knowledge are logged instead of hidden behind a wrong but nearby concept.
- **I5** (learn): Classify disagreements between witness parsers or teachers by cause, so cases of true ambiguity are not used to train repair rules.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 263 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 260 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works · [🔗 Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) · 575 works

## Cite

Cite as: Luisa Bentivogli, Pamela Forner, Claudio Giuliano, Alessandro Marchetti, Emanuele Pianta and Kateryna Tymoshenko. 2010. Extending English ACE 2005 Corpus Annotation with Ground-truth Links to Wikipedia. Proceedings of the 2nd Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bentivogli2010extending,
  title = {{Extending English ACE 2005 Corpus Annotation with Ground-truth Links to Wikipedia}},
  author = {Luisa Bentivogli, Pamela Forner, Claudio Giuliano, Alessandro Marchetti, Emanuele Pianta and Kateryna Tymoshenko},
  booktitle = {Proceedings of the 2nd Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources},
  year = {2010},
  url = {https://aclanthology.org/W10-3503/}
}
```

- [← previous Aligning FrameNet and WordNet based on Semantic Neighborhoods](https://md.research.elamur.ai/free/en/papers/ferrandez-2010-aligning-framenet-wordnet-based-semantic/)
- [next → Transliterating Urdu for a Broad-Coverage Urdu/Hindi LFG Grammar](https://md.research.elamur.ai/free/en/papers/malik-2010-transliterating-urdu-broad-coverage-urdu/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/)

## References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bryl, Volha, Claudio Giuliano, Luciano Serafini, and Kateryna Tymoshenko. 2010. Using background knowledge to support coreference resolution. In Proceedings of the 19th European Conference on Artificial Intelligence (ECAI 2010), August. [find](https://scholar.google.com/scholar?q=Bryl%2C+Volha%2C+Claudio+Giuliano%2C+Luciano+Serafini%2C+and+Kateryna+Tymoshenko%2E+2010%2E+Using+background+knowledge+to+support+coreference+resolution%2E+In+Proceedings+of+the+19th+European+Conference+on+Artifici)
2. Csomai, Andras and Rada Mihalcea. 2008. Linking documents to encyclopedic knowledge. IEEE Intelligent Systems, 23(5):34–41. [find](https://scholar.google.com/scholar?q=Csomai%2C+Andras+and+Rada+Mihalcea%2E+2008%2E+Linking+documents+to+encyclopedic+knowledge%2E+IEEE+Intelligent+Systems%2C+23%285%29%3A34%E2%80%9341%2E)
3. Cucerzan, Silviu. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 708–716, Prague, Czech Republic, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cucerzan%2C+Silviu%2E+2007%2E+Large%2Dscale+named+entity+disambiguation+based+on+Wikipedia+data%2E+In+Proceedings+of+the+2007+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+Computation)
4. Culotta, Aron, Michael L. Wick, and Andrew McCallum. 2007. [First-order probabilistic models for coreference resolution](https://md.research.elamur.ai/free/en/papers/culotta-2007-first-order-probabilistic-models-coreference/). In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 81–88.
5. Fellbaum, Christiane, editor. 1998. WordNet: an electronic lexical database. MIT Press. [find](https://scholar.google.com/scholar?q=Fellbaum%2C+Christiane%2C+editor%2E+1998%2E+WordNet%3A+an+electronic+lexical+database%2E+MIT+Press%2E)
6. Kulkarni, Sayali, Amit Singh, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2009. Collective annotation of wikipedia entities in web text. In KDD ’09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 457–466, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Kulkarni%2C+Sayali%2C+Amit+Singh%2C+Ganesh+Ramakrishnan%2C+and+Soumen+Chakrabarti%2E+2009%2E+Collective+annotation+of+wikipedia+entities+in+web+text%2E+In+KDD+%E2%80%9909%3A+Proceedings+of+the+15th+ACM+SIGKDD+international+c)
7. Mihalcea, Rada and Dan I. Moldovan. 2001. Ez.wordnet: Principles for automatic generation of a coarse grained wordnet. In Russell, Ingrid and John F. Kolen, editors, FLAIRS Conference, pages 454–458. AAAI Press. [find](https://scholar.google.com/scholar?q=Mihalcea%2C+Rada+and+Dan+I%2E+Moldovan%2E+2001%2E+Ez%2Ewordnet%3A+Principles+for+automatic+generation+of+a+coarse+grained+wordnet%2E+In+Russell%2C+Ingrid+and+John+F%2E+Kolen%2C+editors%2C+FLAIRS+Conference%2C+pages+454%E2%80%93458%2E+)
8. Milne, David and Ian H. Witten. 2008. Learning to link with wikipedia. In CIKM ’08: Proceeding of the 17th ACM conference on Information and knowledge management, pages 509–518, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Milne%2C+David+and+Ian+H%2E+Witten%2E+2008%2E+Learning+to+link+with+wikipedia%2E+In+CIKM+%E2%80%9908%3A+Proceeding+of+the+17th+ACM+conference+on+Information+and+knowledge+management%2C+pages+509%E2%80%93518%2C+New+York%2C+NY%2C+USA%2E+ACM)
9. Ng, Vincent and Claire Cardie. 2002. [Improving machine learning approaches to coreference resolution](https://md.research.elamur.ai/free/en/papers/ng-2002-improving-machine-learning-approaches-coreference/). In ACL ’02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 104–111.
10. Ng, Vincent. 2007. [Semantic class induction and coreference resolution](https://md.research.elamur.ai/free/en/papers/ng-2007-semantic-class-induction-coreference-resolution/). In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic, pages 536–543.
11. Ponzetto, S. P. and M. Strube. 2006. [Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution](https://md.research.elamur.ai/free/en/papers/ponzetto-2006-exploiting-semantic-role-labeling-wordnet/). Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 192– 199.
12. Soon, Wee Meng, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. [A machine learning approach to coreference resolution of noun phrases](https://md.research.elamur.ai/free/en/papers/soon-2001-machine-learning-approach-coreference-resolution/). Computational Linguistic, 27(4):521–544.
13. Versley, Yannick, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang, and Alessandro Moschitti. 2008. [Bart: a modular toolkit for coreference resolution](https://md.research.elamur.ai/free/en/papers/versley-2008-bart-modular-toolkit-coreference-resolution/). In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies, pages 9–12.

*Updated 2 November 2026*