--- title: "The stages of event extraction — Ahn 2006" description: "The paper looks at automatic extraction of ACE 2005 events." authors: ["David Ahn"] year: 2006 venue: "Proceedings of the Workshop on Annotating and Reasoning about Time and Events" source: "https://aclanthology.org/W06-0901/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ahn-2006-stages-event-extraction/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ahn-2006-stages-event-extraction/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/) / The stages of event extraction # The stages of event extraction **Authors:** [David Ahn](https://md.research.elamur.ai/free/en/authors/david-ahn/) · **Year:** 2006 · **Venue:** Proceedings of the Workshop on Annotating and Reasoning about Time and Events · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W06-0901/) · [PDF](https://aclanthology.org/W06-0901.pdf) · [OpenAlex](https://openalex.org/W2072628044) **Citations:** 511 citations (openalex-snapshot · 2026-10-14) · [193](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper looks at automatic extraction of ACE 2005 events. These are rich structures: an anchor word, typed arguments, four attributes, and coreference links between mentions. Ahn builds a deliberately simple, modular pipeline that splits the problem into four classification stages: anchor identification, argument identification, attribute assignment and event coreference. Each stage is trained with TiMBL (memory-based) and MegaM (maximum entropy), and MegaM alone is used for coreference because that step needs probabilities. Following the ACE diagnostic setup, gold entities, timexes and values are given as input. Every stage is first evaluated on its own with precision, recall, F-measure or accuracy. Then gold and learned outputs are swapped stage by stage to see how much each one affects the overall ACE value. With all learned components, the system reaches an ACE value of 22.3%, which falls inside the range reported for the 2005 diagnostic task. The ablation shows that argument and anchor errors cost the most, while attributes and coreference matter much less. This gives a clear guide to where research effort pays off. ## Key points - Data: the ACE 2005 training corpus (599 documents), split by document into 539 for training and 60 for testing. Another 60 training documents are held out as a development set for MegaM tuning. Sentences are parsed with the Charniak parser and the parses are converted to dependencies. - Anchor identification is framed as word-level classification into 34 classes (33 event types plus None). This works because over 95% of anchors are a single word. The class skew is extreme: 197,261 of 202,135 training instances are None, and the largest real class, Conflict:Attack, has only 1,410. - Modality, polarity and genericity barely beat the majority baseline. For tense, MegaM reaches 0.633 accuracy against a baseline of 0.535. The event coreference pair classifier gets F = 0.658, while a same-type baseline gets 0.286. - In the ACE value ablation, replacing gold arguments with learned ones costs 35.6 points (81.1% to 45.5%). Learned anchors cost about 21–22 points. Learned attributes cost 4.8 points relative to gold attributes. - Perfect coreference adds 7.5 points over no coreference when the other stages use gold data (81.1% to 88.6%). The learned coreference classifier adds only 0.6 points in the fully learned pipeline. ## Interesting ideas - **Extending gold-input diagnostics to each stage.** Feeding every component the gold output of the stages before it shows each step's true difficulty, separate from propagated errors. This could help profile any multi-stage NLP pipeline, such as parsing followed by NER followed by relation extraction. - **A gold/learned swap matrix scored with the end-task metric.** Running the full system under many gold/learned/majority combinations shows which stage limits final performance. Dependencies between stages mean not every combination is valid. The result helps decide where to spend annotation or modelling effort. - **Majority-class components as a sanity check.** Putting a trivial classifier in a slot can reveal that a "learned" module adds almost nothing to the final score, as happened here with the attributes. This is a cheap test worth running before investing in a component. - **Layered fallback for aligning mentions to parses.** The aligner first looks for an exact constituent, then a sequence of constituents, then a word matching the mention's start or end, and finally a word containing the whole mention. This makes independently produced annotations usable with a parser's output, which is common when combining corpora with different tokenisation. - **Coreference features built from argument overlap.** Two event mentions are compared through the arguments they share, the arguments they don't share, and shared arguments that appear in different roles. This structural similarity signal could carry over to linking other predicate-argument structures, for example relations or frames. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) ## Cited by Cited by 193 papers of our library cite it (of the 140884 whose bibliographies we read); 24 of them have a page here, 169 not yet. - [Verb Phrase Ellipsis Resolution Using Discriminative and Margin-Infused Algorithms](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-dean-verb-phrase-ellipsis-resolution-discriminative/) - [Unsupervised Event Coreference Resolution with Rich Linguistic Features](https://md.research.elamur.ai/free/en/papers/bejan-2010-unsupervised-event-coreference-resolution-rich/) - [Unsupervised Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/bejan-2014-unsupervised-event-coreference-resolution/) - [A Pairwise Event Coreference Model, Feature Impact and Evaluation for Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/chen-2009-pairwise-event-coreference-model-feature/) - [Chinese Event Coreference Resolution: An Unsupervised Probabilistic Model Rivaling Supervised Resolvers](https://md.research.elamur.ai/free/en/papers/chen-2015-chinese-event-coreference-resolution-unsupervised/) - [Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/chen-2015-event-extraction-dynamic-multi-pooling/) - [Event Coreference Resolution by Iteratively Unfolding Inter-dependencies among Events](https://md.research.elamur.ai/free/en/papers/choubey-2017-event-coreference-resolution-iteratively-unfolding/) - [Improving Event Coreference Resolution by Modeling Correlations between Event Coreference Chains and Document Topic Structures](https://md.research.elamur.ai/free/en/papers/choubey-2018-improving-event-coreference-resolution-modeling/) - [Using Cross-Entity Inference to Improve Event Extraction](https://md.research.elamur.ai/free/en/papers/hong-2011-cross-entity-inference-improve-event/) - [DEGREE: A Data-Efficient Generation-Based Event Extraction Model](https://md.research.elamur.ai/free/en/papers/hsu-2022-degree-data-efficient-generation-based/) - [Refining Event Extraction through Cross-Document Inference](https://md.research.elamur.ai/free/en/papers/ji-2008-refining-event-extraction-through-cross/) - [Joint Event Extraction via Structured Prediction with Global Features](https://md.research.elamur.ai/free/en/papers/li-2013-joint-event-extraction-structured-prediction/) - [Using Document Level Cross-Event Inference to Improve Event Extraction](https://md.research.elamur.ai/free/en/papers/liao-2010-document-level-cross-event-inference/) - [Supervised Within-Document Event Coreference using Information Propagation](https://md.research.elamur.ai/free/en/papers/liu-2014-supervised-within-document-event-coreference/) - [Leveraging FrameNet to Improve Automatic Event Detection](https://md.research.elamur.ai/free/en/papers/liu-2016-leveraging-framenet-improve-automatic-event/) - [Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms](https://md.research.elamur.ai/free/en/papers/liu-2017-exploiting-argument-information-improve-event/) - [Event Extraction as Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/liu-2020-event-extraction-as-machine-reading/) - [Joint Inference for Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/lu-2016-joint-inference-event-coreference-resolution/) - [Conundrums in Event Coreference Resolution: Making Sense of the State of the Art](https://md.research.elamur.ai/free/en/papers/lu-2021-conundrums-event-coreference-resolution-making/) - [Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction](https://md.research.elamur.ai/free/en/papers/lu-2021-text2event-controllable-sequence-structure-generation/) - [Joint Event Extraction via Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2016-joint-event-extraction-recurrent-neural/) - [Exploiting Document Structures and Cluster Consistencies for Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/tran-2021-exploiting-document-structures-cluster-consistencies/) - [MAVEN: A Massive General Domain Event Detection Dataset](https://md.research.elamur.ai/free/en/papers/wang-2020-maven-massive-general-domain-event/) - [Joint Extraction of Events and Entities within a Document Context](https://md.research.elamur.ai/free/en/papers/yang-2016-joint-extraction-events-entities-within/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ahn2006stages, title = {{The stages of event extraction}}, author = {Ahn, David}, booktitle = {Proceedings of the Workshop on Annotating and Reasoning about Time and Events}, year = {2006}, doi = {10.3115/1629235.1629236}, url = {https://doi.org/10.3115/1629235.1629236} } ``` - [← previous The Second Release of the RASP System](https://md.research.elamur.ai/free/en/papers/briscoe-2006-second-release-rasp-system/) - [next → The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/levow-2006-third-international-chinese-language-processing/) - [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/) ## References 9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 2005. The ACE 2005 (ACE05) evaluation plan. [http://www.nist.gov/speech/tests/ace/ace05/doc/ace05-evalplan.v3](http://www.nist.gov/speech/tests/ace/ace05/doc/ace05-evalplan.v3). pdf. [↗](http://www.nist.gov/speech/tests/ace/ace05/doc/ace05-evalplan.v3) 2. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Meeting of NAACL, pages 132–139. [find](https://scholar.google.com/scholar?q=Eugene+Charniak%2E+2000%2E+A+maximum%2Dentropy%2Dinspired+parser%2E+In+Proceedings+of+the+1st+Meeting+of+NAACL%2C+pages+132%E2%80%93139%2E) [our card](https://md.research.elamur.ai/free/en/papers/charniak-2000-maximum-entropy-inspired-parser/) 3. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+1999%2E+Head%2DDriven+Statistical+Models+for+Natural+Language+Parsing%2E+Ph%2ED%2E+thesis%2C+University+of+Pennsylvania%2E) 4. Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2004. TiMBL: Tilburg Memory Based Learner, version 5.1, Reference Guide. University of Tilburg, ILK Technical Report ILK-0402. [http://ilk.uvt.nl/](http://ilk.uvt.nl/). [↗](http://ilk.uvt.nl/) 5. Hal Daumé III. 2004. Notes on CG and LM-BFGS optimization of logistic regression. Paper available at [http://www.isi.edu/˜hdaume/docs/daume04cg-bfgs.ps](http://www.isi.edu/˜hdaume/docs/daume04cg-bfgs.ps), August. [↗](http://www.isi.edu/˜hdaume/docs/daume04cg-bfgs.ps) 6. Radu Florian, Hany Hassan, Abraham Ittycheriah, Hongyan Jing, Nanda Kambhatla, Xiaoqiang Luo, Nicolas Nicolov, and Salim Roukos. 2004. A statistical model for multilingual entity detection and tracking. In Proceedings of HLT/NAACL-04. [find](https://scholar.google.com/scholar?q=Radu+Florian%2C+Hany+Hassan%2C+Abraham+Ittycheriah%2C+Hongyan+Jing%2C+Nanda+Kambhatla%2C+Xiaoqiang+Luo%2C+Nicolas+Nicolov%2C+and+Salim+Roukos%2E+2004%2E+A+statistical+model+for+multilingual+entity+detection+and+trackin) 7. Gregory Grefenstette. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer. [find](https://scholar.google.com/scholar?q=Gregory+Grefenstette%2E+1994%2E+Explorations+in+Automatic+Thesaurus+Discovery%2E+Kluwer%2E) 8. Valentin Jijkoun and Maarten de Rijke. 2004. Enriching the output of a parser using memory-based learning. In Proceedings of the 42nd Meeting of the ACL. [find](https://scholar.google.com/scholar?q=Valentin+Jijkoun+and+Maarten+de+Rijke%2E+2004%2E+Enriching+the+output+of+a+parser+using+memory%2Dbased+learning%2E+In+Proceedings+of+the+42nd+Meeting+of+the+ACL%2E) 9. Linguistic Data Consortium, 2005. ACE (Automatic Content Extraction) English Annotation Guidelines for Events, version 5.4.3 2005.07.01 edition. [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2C+2005%2E+ACE+%28Automatic+Content+Extraction%29+English+Annotation+Guidelines+for+Events%2C+version+5%2E4%2E3+2005%2E07%2E01+edition%2E)