--- title: "Using Document Level Cross-Event Inference to — Liao 2010" description: "Most ACE event extraction systems decide about triggers and arguments one sentence at a time, and local context often cannot resolve the ambiguity." authors: ["Shasha Liao", "Ralph Grishman"] year: 2010 venue: "Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P10-1081/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/liao-2010-document-level-cross-event-inference/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liao-2010-document-level-cross-event-inference/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) / Using Document Level Cross-Event Inference to Improve Event Extraction # Using Document Level Cross-Event Inference to Improve Event Extraction **Authors:** Shasha Liao, [Ralph Grishman](https://md.research.elamur.ai/free/en/authors/ralph-grishman/) · **Year:** 2010 · **Venue:** Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P10-1081/) · [PDF](https://aclanthology.org/P10-1081.pdf) · [OpenAlex](https://openalex.org/W2108743083) **Citations:** 278 citations (openalex-snapshot · 2026-10-14) · [129](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Most ACE event extraction systems decide about triggers and arguments one sentence at a time, and local context often cannot resolve the ambiguity. For example, "left" can be a Transport event or an End-Position event. Liao and Grishman use evidence from the whole document, including events of *other* types, not just repeated events of the same type. Their two-stage approach works as follows: 1. A sentence-level MaxEnt baseline tags events. 2. Only high-confidence decisions are kept in a document-level table of confident events, with a separate table for conflicting role assignments. 3. Two extra MaxEnt classifiers, one for triggers and one for arguments, use features describing which other event types and entity roles appear elsewhere in the document. They re-tag words and entity mentions that were left uncertain. Training used ACE 2005 newswire data. On a blind test set, the method improves F-score by 9.0 points for trigger classification, 9.0 for argument classification and 8.1 for role classification over the baseline. It also clearly beats a reproduction of Ji and Grishman's (2008) within-event-type consistency rules. The work shows that typical co-occurrence of event types is a cheap but strong signal for information extraction. ## Key points - **Task:** ACE 2005 event extraction with 8 types and 33 subtypes. The subtypes are treated as flat, independent classes, and event mention coreference is ignored. - **Data split:** - 549 ACE 2005 training documents train both the baseline and the document-level taggers. - 10 newswire texts form the development set. - 40 newswire texts form the blind test set, following Ji and Grishman (2008). - **Trigger F-score:** 59.74 for the baseline, 61.43 with within-event-type rules, and 68.79 with the cross-event model. - **Argument and role F-scores:** | System | Argument F | Role F | |---|---|---| | Baseline | 41.29 | 36.46 | | Within-event-type rules | 47.35 | 42.21 | | Cross-event model | 50.28 | 44.55 | Gains come mostly from recall, though precision also rises. - **Human comparison:** two annotators were evaluated on 28 of the test texts. - Their trigger F-scores were 59.3 and 72.0, so the system's 68.79 falls between them. - Their role F-scores were 55.3 and 62.4, which remain above the system's 44.55. - This low agreement highlights how difficult the task is. ## Interesting ideas - **Tag easy cases first, then use them as context.** Unambiguous triggers such as "appointment" or "founded" can help decide hard ones like "stepped" (down). This pattern suits any document-level labeling task where some decisions are much more certain than others. - **Learn inter-type statistics instead of hand-writing them.** Co-occurrence of event types and role correspondences (e.g., an entity filling the Victim role in a Die event frequently also fills the Target role in an Attack) can be estimated from the training corpus and fed in as classifier features. This avoids costly hand-built inference rules and could transfer to relation or frame extraction in other languages. - **Build features from conjunctions.** Each feature combines the event type being predicted, another event type in the document, and the role the same entity plays for that other type. Combined with coreference, this lets an easily found argument in one sentence support a hard, distant one elsewhere. - **Keep only confident decisions and track conflicts.** Low-confidence tags are filtered out, and a separate table resolves an entity that receives contradictory roles. This limits error propagation when first-pass outputs become features for a second pass. - **Report human agreement next to system scores.** Scoring two individual annotators against the adjudicated key gives a realistic ceiling and shows how hard the task is. This is useful when creating new annotated resources, for example Ukrainian event corpora. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 129 papers of our library cite it (of the 140884 whose bibliographies we read); 18 of them have a page here, 111 not yet. - [Joint Event Trigger Identification and Event Coreference Resolution with Structured Perceptron](https://md.research.elamur.ai/free/en/papers/araki-2015-joint-event-trigger-identification-event/) - [Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/chen-2015-event-extraction-dynamic-multi-pooling/) - [A Language-Independent Neural Network for Event Detection](https://md.research.elamur.ai/free/en/papers/feng-2016-language-independent-neural-network-event/) - [Richer Event Description: Integrating event coreference with temporal, causal and bridging annotation](https://md.research.elamur.ai/free/en/papers/gorman-2016-richer-event-description-integrating-event/) - [Using Cross-Entity Inference to Improve Event Extraction](https://md.research.elamur.ai/free/en/papers/hong-2011-cross-entity-inference-improve-event/) - [Zero-Shot Transfer Learning for Event Extraction](https://md.research.elamur.ai/free/en/papers/huang-2018-zero-shot-transfer-learning-event/) - [Joint Event Extraction via Structured Prediction with Global Features](https://md.research.elamur.ai/free/en/papers/li-2013-joint-event-extraction-structured-prediction/) - [Event Extraction as Multi-turn Question Answering](https://md.research.elamur.ai/free/en/papers/li-2020-event-extraction-as-multi-turn/) - [Leveraging FrameNet to Improve Automatic Event Detection](https://md.research.elamur.ai/free/en/papers/liu-2016-leveraging-framenet-improve-automatic-event/) - [Exploiting Argument Information to Improve Event Detection via Supervised Attention Mechanisms](https://md.research.elamur.ai/free/en/papers/liu-2017-exploiting-argument-information-improve-event/) - [Jointly Multiple Events Extraction via Attention-based Graph Information Aggregation](https://md.research.elamur.ai/free/en/papers/liu-2018-jointly-multiple-events-extraction-attention/) - [Event Extraction as Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/liu-2020-event-extraction-as-machine-reading/) - [Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction](https://md.research.elamur.ai/free/en/papers/lu-2021-text2event-controllable-sequence-structure-generation/) - [Event Detection and Domain Adaptation with Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2015-event-detection-domain-adaptation-convolutional/) - [Joint Event Extraction via Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/nguyen-2016-joint-event-extraction-recurrent-neural/) - [Event Detection and Co-reference with Minimal Supervision](https://md.research.elamur.ai/free/en/papers/peng-2016-event-detection-co-reference-minimal/) - [Joint Extraction of Events and Entities within a Document Context](https://md.research.elamur.ai/free/en/papers/yang-2016-joint-extraction-events-entities-within/) - [Exploring Pre-trained Language Models for Event Extraction and Generation](https://md.research.elamur.ai/free/en/papers/yang-2019-exploring-pre-trained-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{liao2010using, title = {{Using Document Level Cross-Event Inference to Improve Event Extraction}}, author = {Liao, Shasha and Grishman, Ralph}, booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics}, year = {2010}, url = {https://aclanthology.org/P10-1081/} } ``` - [← previous Using an Error-Annotated Learner Corpus to Develop an ESL/EFL Error Correction System](https://md.research.elamur.ai/free/en/papers/han-2010-error-annotated-learner-corpus-develop/) - [next → Using entity features to classify implicit discourse relations](https://md.research.elamur.ai/free/en/papers/louis-2010-entity-features-classify-implicit-discourse/) - [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) ## References 12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. David Ahn. 2006. The stages of event extraction. In Proc. COLING/ACL 2006 Workshop on Annotating and Reasoning about Time and Events. Sydney, Australia. [find](https://scholar.google.com/scholar?q=David+Ahn%2E+2006%2E+The+stages+of+event+extraction%2E+In+Proc%2E+COLING%2FACL+2006+Workshop+on+Annotating+and+Reasoning+about+Time+and+Events%2E+Sydney%2C+Australia%2E) [our card](https://md.research.elamur.ai/free/en/papers/ahn-2006-stages-event-extraction/) 2. J. Finkel, T. Grenager, and C. Manning. 2005. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In Proc. 43rd Annual Meeting of the Association for Computational Linguistics, pages 363–370, Ann Arbor, MI, June. [find](https://scholar.google.com/scholar?q=J%2E+Finkel%2C+T%2E+Grenager%2C+and+C%2E+Manning%2E+2005%2E+Incorporating+Non%2Dlocal+Information+into+Information+Extraction+Systems+by+Gibbs+Sampling%2E+In+Proc%2E+43rd+Annual+Meeting+of+the+Association+for+Computation) [our card](https://md.research.elamur.ai/free/en/papers/finkel-2005-incorporating-non-local-information-into/) 3. Ralph Grishman, David Westbrook and Adam Meyers. 2005. [find](https://scholar.google.com/scholar?q=Ralph+Grishman%2C+David+Westbrook+and+Adam+Meyers%2E+2005%2E) 4. NYU’s English ACE 2005 System Description. In Proc. [find](https://scholar.google.com/scholar?q=NYU%E2%80%99s+English+ACE+2005+System+Description%2E+In+Proc%2E) 5. ACE 2005 Evaluation Workshop, Gaithersburg, MD. [find](https://scholar.google.com/scholar?q=ACE+2005+Evaluation+Workshop%2C+Gaithersburg%2C+MD%2E) 6. Prashant Gupta, Heng Ji. 2009. Predicting Unknown Time Arguments based on Cross-Event Propagation. In Proc. [find](https://scholar.google.com/scholar?q=Prashant+Gupta%2C+Heng+Ji%2E+2009%2E+Predicting+Unknown+Time+Arguments+based+on+Cross%2DEvent+Propagation%2E+In+Proc%2E) 7. Hilda Hardy, Vika Kanchakouskaya and Tomek Strzalkowski. 2006. Automatic Event Classification Using Surface Text Features. In Proc. AAAI06 Workshop on Event Extraction and Synthesis. Boston, MA. [find](https://scholar.google.com/scholar?q=Hilda+Hardy%2C+Vika+Kanchakouskaya+and+Tomek+Strzalkowski%2E+2006%2E+Automatic+Event+Classification+Using+Surface+Text+Features%2E+In+Proc%2E+AAAI06+Workshop+on+Event+Extraction+and+Synthesis%2E+Boston%2C+MA%2E) 8. H. Ji and R. Grishman. 2008. Refining Event Extraction through Cross-Document Inference. In Proc. ACL-08: HLT, pages 254–262, Columbus, OH, June. [find](https://scholar.google.com/scholar?q=H%2E+Ji+and+R%2E+Grishman%2E+2008%2E+Refining+Event+Extraction+through+Cross%2DDocument+Inference%2E+In+Proc%2E+ACL%2D08%3A+HLT%2C+pages+254%E2%80%93262%2C+Columbus%2C+OH%2C+June%2E) [our card](https://md.research.elamur.ai/free/en/papers/ji-2008-refining-event-extraction-through-cross/) 9. M. Maslennikov and T. Chua. 2007. A Multi resolution Framework for Information Extraction from Free Text. In Proc. 45th Annual Meeting of the Association of Computational Linguistics, pages 592–599, Prague, Czech Republic, June. [find](https://scholar.google.com/scholar?q=M%2E+Maslennikov+and+T%2E+Chua%2E+2007%2E+A+Multi+resolution+Framework+for+Information+Extraction+from+Free+Text%2E+In+Proc%2E+45th+Annual+Meeting+of+the+Association+of+Computational+Linguistics%2C+pages+592%E2%80%93599%2C+P) 10. S. Patwardhan and E. Riloff. 2007. Effective Information Extraction with Semantic Affinity Patterns and Relevant Regions. In Proc. Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 2007, pages 717–727, Prague, Czech Republic, June. [find](https://scholar.google.com/scholar?q=S%2E+Patwardhan+and+E%2E+Riloff%2E+2007%2E+Effective+Information+Extraction+with+Semantic+Affinity+Patterns+and+Relevant+Regions%2E+In+Proc%2E+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) 11. Patwardhan, S. and Riloff, E. 2009. A Unified Model of Phrasal and Sentential Evidence for Information Extraction. In Proc. Conference on Empirical Methods in Natural Language Processing 2009, (EMNLP-09). [find](https://scholar.google.com/scholar?q=Patwardhan%2C+S%2E+and+Riloff%2C+E%2E+2009%2E+A+Unified+Model+of+Phrasal+and+Sentential+Evidence+for+Information+Extraction%2E+In+Proc%2E+Conference+on+Empirical+Methods+in+Natural+Language+Processing+2009%2C+%28EMNLP%2D) 12. Unsupervised Word Sense Disambiguation Rivaling Supervised Methods. In Proc. ACL 1995. Cambridge, MA. [find](https://scholar.google.com/scholar?q=Unsupervised+Word+Sense+Disambiguation+Rivaling+Supervised+Methods%2E+In+Proc%2E+ACL+1995%2E+Cambridge%2C+MA%2E) [our card](https://md.research.elamur.ai/free/en/papers/yarowsky-1995-unsupervised-word-sense-disambiguation-rivaling/)