--- title: "A Comparison of the Events and Relations — Aguilar 2014" description: "Work on relation extraction and semantic parsing depends on several hand-annotated resources." authors: ["Jacqueline Aguilar", "Charley Beller", "Paul McNamee", "Benjamin Van Durme", "Stephanie Strassel", "Zhiyi Song", "Joe Ellis"] year: 2014 venue: "Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation" source: "https://aclanthology.org/W14-2907/" doi: "10.3115/v1/w14-2907" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/aguilar-2014-comparison-events-relations-across-ace/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/aguilar-2014-comparison-events-relations-across-ace/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2014](https://md.research.elamur.ai/free/papers/2014/) / A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards # A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards **Authors:** Jacqueline Aguilar, Charley Beller, [Paul McNamee](https://md.research.elamur.ai/free/authors/paul-mcnamee/), [Benjamin Van Durme](https://md.research.elamur.ai/free/authors/benjamin-van-durme/), [Stephanie Strassel](https://md.research.elamur.ai/free/authors/stephanie-strassel/), [Zhiyi Song](https://md.research.elamur.ai/free/authors/zhiyi-song/), Joe Ellis · **Year:** 2014 · **Venue:** Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W14-2907/) · [PDF](https://aclanthology.org/W14-2907.pdf) · [DOI 10.3115/v1/w14-2907](https://doi.org/10.3115/v1/w14-2907) · [OpenAlex](https://openalex.org/W2250576901) **Citations:** 90 citations (openalex-snapshot · 2026-10-14) · [46](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Work on relation extraction and semantic parsing depends on several hand-annotated resources. These were built under different guidelines and for different purposes, which makes them hard to reuse together. This paper, a collaboration between the JHU HLTCOE and the Linguistic Data Consortium, surveys four standards side by side: ACE, its simplified successor ERE, TAC-KBP Slot-filling, and FrameNet. The authors first show how entity-level decisions shape relation and event annotation. These decisions include type inventories, subtypes, entity classes and mention extents. They then contrast ACE and ERE on relation types, argument constraints, modality, tense, syntactic classes and triggers. Next, they outline the narrow, question-oriented TAC-KBP slot schema. Finally, they map relations, events and attributes onto FrameNet frames. The main conclusions are: - ERE gives up some of ACE's granularity to gain annotator consistency. - TAC-KBP targets entity slots and does not model events explicitly. - FrameNet is the most expressive scheme. Content from the other standards can generally be mapped into it, provided type and argument granularity are adjusted. The survey helps practitioners align, compare and convert annotated corpora. ## Key points - ACE was created by NIST in 1999, and its last evaluation took place in 2008. ERE was designed under DARPA DEFT as a lighter variant with these changes: - Facility and Location are merged into Location. - Vehicle and Weapon are dropped, and a Title type is added. - Subtypes are removed. - Only Specific mentions are tagged. - ERE annotates only asserted relations. ACE also covers hypothetical ones through Modality (ASSERTED/OTHER) and four Tense values. ACE additionally uses Syntactic Classes and clause-level spans, whereas ERE uses an optional minimal trigger. - The relation inventories differ: - ERE adds Physical.Origin, Social.Membership and Social.Role. - ERE lacks ACE's Near and Part-Whole.Geographical subtypes. - ERE has only two Affiliation subtypes, while ACE has many. - FrameNet 1.5 contains 1020 frames and 11830 lexical units, and mappings to it can be one-to-many: - ERE Affiliation.Employment/Membership corresponds to both Being Employed and Membership. - The Kinship frame corresponds to five TAC-KBP slots. - The core elements of the Extradition frame match ERE's Justice-Extradition arguments exactly. ACE's extra Crime and Time arguments map to non-core elements. - The TAC-KBP Cold Start task, introduced in 2012, builds knowledge bases from collections of roughly O(50,000) documents. These KBs are evaluated by sampling. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Semantic role labeling](https://md.research.elamur.ai/free/topics/semantic-role-labeling/) ## Cited by Cited by 46 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 44 not yet. - [Cross-Document, Cross-Language Event Coreference Annotation Using Event Hoppers](https://md.research.elamur.ai/free/papers/song-2018-cross-document-cross-language-event/) - [MAVEN: A Massive General Domain Event Detection Dataset](https://md.research.elamur.ai/free/papers/wang-2020-maven-massive-general-domain-event/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{aguilar2014comparison, title = {{A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards}}, author = {Aguilar, Jacqueline and Beller, Charley and McNamee, Paul and Durme, Benjamin Van and Strassel, Stephanie and Song, Zhiyi and Ellis, Joe}, booktitle = {Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation}, year = {2014}, doi = {10.3115/v1/w14-2907}, url = {https://doi.org/10.3115/v1/w14-2907} } ``` - [← previous A Bayesian Mixed Effects Model of Literary Character](https://md.research.elamur.ai/free/papers/bamman-2014-bayesian-mixed-effects-model-literary/) - [next → A Constituent-Based Approach to Argument Labeling with Joint Inference in Discourse Parsing](https://md.research.elamur.ai/free/papers/kong-2014-constituent-based-approach-argument-labeling/) - [↑ back to papers of 2014](https://md.research.elamur.ai/free/papers/2014/) ## References 15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 86–90. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Collin+F+Baker%2C+Charles+J+Fillmore%2C+and+John+B+Lowe%2E+1998%2E+The+berkeley+framenet+project%2E+In+Proceedings+of+the+17th+international+conference+on+Computational+linguistics%2DVolume+1%2C+pages+86%E2%80%9390%2E+Associ) [our card](https://md.research.elamur.ai/free/papers/lex-1998-baker-berkeley-framenet/) 2. Collin Baker, Michael Ellsworth, and Katrin Erk. 2007. Semeval-2007 task 19: Frame semantic structure extraction. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 99–104, Prague, Czech Republic, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Collin+Baker%2C+Michael+Ellsworth%2C+and+Katrin+Erk%2E+2007%2E+Semeval%2D2007+task+19%3A+Frame+semantic+structure+extraction%2E+In+Proceedings+of+the+Fourth+International+Workshop+on+Semantic+Evaluations+%28SemEval%2D2) [our card](https://md.research.elamur.ai/free/papers/baker-2007-semeval-2007-task-19-frame/) 3. Dipanjan Das, Nathan Schneider, Desai Chen, and Noah A Smith. 2010. Probabilistic frame-semantic parsing. In Proceedings of NAACL-HLT, pages 948– 956. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dipanjan+Das%2C+Nathan+Schneider%2C+Desai+Chen%2C+and+Noah+A+Smith%2E+2010%2E+Probabilistic+frame%2Dsemantic+parsing%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+948%E2%80%93+956%2E+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/papers/das-2010-probabilistic-frame-semantic-parsing/) 4. George Doddington, Alexis Mitchell, Mark Przybocki, Lancec Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ace) program- tasks, data, and evaluation. In Proceedings of LREC 2004: Fourth International Conference on Language Resources and Evaluation, Lisbon,… [find](https://scholar.google.com/scholar?q=George+Doddington%2C+Alexis+Mitchell%2C+Mark+Przybocki%2C+Lancec+Ramshaw%2C+Stephanie+Strassel%2C+and+Ralph+Weischedel%2E+2004%2E+The+automatic+content+extraction+%28ace%29+program%2D+tasks%2C+data%2C+and+evaluation%2E+In+Proc) [our card](https://md.research.elamur.ai/free/papers/doddington-2004-automatic-content-extraction-ace-program/) 5. Charles J Fillmore. 1976. Frame semantics and the nature of language. Annals of the New York Academy of Sciences, 280(1):20–32. [find](https://scholar.google.com/scholar?q=Charles+J+Fillmore%2E+1976%2E+Frame+semantics+and+the+nature+of+language%2E+Annals+of+the+New+York+Academy+of+Sciences%2C+280%281%29%3A20%E2%80%9332%2E) 6. Charles Fillmore. 1982. Frame semantics. In Linguistics in the morning calm, pages 111–137. Hanshin Publishing Co. [find](https://scholar.google.com/scholar?q=Charles+Fillmore%2E+1982%2E+Frame+semantics%2E+In+Linguistics+in+the+morning+calm%2C+pages+111%E2%80%93137%2E+Hanshin+Publishing+Co%2E) 7. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational linguistics, 28(3):245–288. [find](https://scholar.google.com/scholar?q=Daniel+Gildea+and+Daniel+Jurafsky%2E+2002%2E+Automatic+labeling+of+semantic+roles%2E+Computational+linguistics%2C+28%283%29%3A245%E2%80%93288%2E) [our card](https://md.research.elamur.ai/free/papers/gildea-2002-automatic-labeling-semantic-roles/) 8. Linguistic Data Consortium. 2005. ACE (automatic content extraction) English annotation guidelines for events. [https://www.ldc.upenn.edu/collaborations/past-projects/ace](https://www.ldc.upenn.edu/collaborations/past-projects/ace). Version 5.4.3 2005.07.01. [↗](https://www.ldc.upenn.edu/collaborations/past-projects/ace) 9. Linguistic Data Consortium. 2006. ACE (automatic content extraction) English annotation guidelines for entities. [https://www.ldc.upenn.edu/collaborations/past-projects/ace](https://www.ldc.upenn.edu/collaborations/past-projects/ace), Version 5.6.6 2006.08.01. [↗](https://www.ldc.upenn.edu/collaborations/past-projects/ace) 10. Linguistic Data Consortium. 2008. ACE (automatic content extraction) English annotation guidelines for relations. [https://www.ldc.upenn.edu/collaborations/past-projects/ace](https://www.ldc.upenn.edu/collaborations/past-projects/ace). Version 6.0 2008.01.07. [↗](https://www.ldc.upenn.edu/collaborations/past-projects/ace) 11. Linguistic Data Consortium. 2013a. DEFT ERE annotation guidelines: Entities v1.1, 05.17.2013. [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+2013a%2E+DEFT+ERE+annotation+guidelines%3A+Entities+v1%2E1%2C+05%2E17%2E2013%2E) 12. Linguistic Data Consortium. 2013b. DEFT ERE annotation guidelines: Events v1.1. 05.17.2013. [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+2013b%2E+DEFT+ERE+annotation+guidelines%3A+Events+v1%2E1%2E+05%2E17%2E2013%2E) 13. Linguistic Data Consortium. 2013c. DEFT ERE annotation guidelines: Relations v1.1. 05.17.2013. [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+2013c%2E+DEFT+ERE+annotation+guidelines%3A+Relations+v1%2E1%2E+05%2E17%2E2013%2E) 14. Ken Litkowski. 2004. Senseval-3 task: Automatic labeling of semantic roles. In Rada Mihalcea and Phil Edmonds, editors, Senseval-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 9–12, Barcelona, Spain, July. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Ken+Litkowski%2E+2004%2E+Senseval%2D3+task%3A+Automatic+labeling+of+semantic+roles%2E+In+Rada+Mihalcea+and+Phil+Edmonds%2C+editors%2C+Senseval%2D3%3A+Third+International+Workshop+on+the+Evaluation+of+Systems+for+the+Se) 15. Paul McNamee, Hoa Trang Dang, Heather Simpson, Patrick Schone, and Stephanie Strassel. 2010. An evaluation of technologies for knowledge base population. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Paul+McNamee%2C+Hoa+Trang+Dang%2C+Heather+Simpson%2C+Patrick+Schone%2C+and+Stephanie+Strassel%2E+2010%2E+An+evaluation+of+technologies+for+knowledge+base+population%2E+In+Proceedings+of+LREC%2E)