---
title: "A Dataset for Open Event Extraction in English — Nguyen 2016"
description: "The authors argue that MUC-4 is a poor fit for this. 15 references, cited by 6 papers here. Among them: 12."
authors: ["Kiem-Hieu Nguyen", "Xavier Tannier", "Olivier Ferret", "Romaric Besançon"]
year: 2016
venue: "Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)"
source: "https://aclanthology.org/L16-1307/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2016-dataset-open-event-extraction-english/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2016-dataset-open-event-extraction-english/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / A Dataset for Open Event Extraction in English

# A Dataset for Open Event Extraction in English

**Authors:** Kiem-Hieu Nguyen, [Xavier Tannier](https://md.research.elamur.ai/free/en/authors/xavier-tannier/), [Olivier Ferret](https://md.research.elamur.ai/free/en/authors/olivier-ferret/) and Romaric Besançon · **Year:** 2016 · **Venue:** [Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2016](https://md.research.elamur.ai/free/en/venues/lrec/2016/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L16-1307/) · [PDF](https://aclanthology.org/L16-1307.pdf) · [OpenAlex](https://openalex.org/W2572149427)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · 6 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Research on unsupervised event schema induction, which learns event templates and groups entities into roles without labels, has been evaluated almost entirely on MUC-4. The authors argue that MUC-4 is a poor fit for this. It is small and has no redundancy, meaning there are no multiple reports of one event. Its templates also share nearly the same slots, so systems never have to tell event types apart. To fix this they built ASTRE, a partially annotated English corpus. It has 100 Wikinews articles from the Law & Justice category, hand-annotated with TAC KBP 2014 event types, arguments and entity coreference chains. These are paired with about a thousand unannotated web articles about the same stories, collected through Google for unsupervised training. The authors tested two leading systems on the corpus, one from Chambers (2013) and the other from Nguyen et al. (2015). Both scored far below their MUC-4 results, which shows the task is harder than the old benchmark suggested. The resource matters because it gives event-type-specific roles and redundant news coverage, and these better match how modern clustering-based induction methods work.

## Key points
- Annotation covers a subset of TAC KBP 2014 event types: LIFE.{Injure, Die}, CONFLICT.Attack, and nine JUSTICE subtypes (Charge-Indict, Arrest-Jail, Sentence, Convict, Appeal, Acquit, Execute, Extradite, Release-Parole). The tool was Brat.
- The 100 annotated documents (one event each) were hand-picked from roughly 4,000 Law & Justice Wikinews articles. Coreference chains were marked only for entities that serve as an event argument at least once.
- Unannotated data was retrieved by sending each article title to Google and keeping up to K=20 hits whose creation date fell within ±7 days of the seed article.
- The cleaning pipeline went 1,724 links → 1,347 reachable → 1,186 after boilerpipe extraction and dropping files under 1KB → 1,038 after near-duplicate removal with SpotSigs plus Markov Clustering (mcl).
- The final unannotated corpus has about 42.6K sentences, 969.5K words and 1.19M tokens. Chambers (2013) showed results strongly skewed toward recall, while Nguyen et al. (2015) had balanced precision and recall.

## Interesting ideas
- Annotate coreference only for entities that fill an event role, and skip event-event coreference entirely, which cuts cost while keeping what role clustering needs.
- Gather redundant unlabeled text by using a seed article's title as a search query and filtering hits to a narrow date window, which yields several independent retellings of one event.
- Accept a mix of relevant and unrelated documents in the training pool on purpose, treating that noise as realism instead of a defect.
- Remove leftover boilerplate by ranking the most frequent lines across the whole corpus and reviewing only the top 200 by hand.
- A benchmark whose templates share nearly identical slots lets systems score well without separating event types, so evaluation sets should force role distinctions that depend on the type.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): Our coreference gold data could be limited to mentions that fill VerbNet roles in frames, so annotation effort goes only to links that change who-did-what-to-whom.
- **I2** (learn): Parallel retellings of one story give a natural agreement check, since frames extracted from different versions should match, offering a cheap silver-data signal beyond witness-parser votes.
- **I3** (learn): Unrelated documents mixed into a story cluster can serve as built-in negative controls when checking whether induced event structures really belong to a story.
- **I4** (tokens): A cheap frequency-plus-manual-review pass could strip navigation or footer junk from scraped English text before tokenization and tagging.
- **I5** (world): Tests of event ordering and states should include event types with different role sets (e.g., Sentence vs. Attack), so the world model must keep them apart and not merely match generic agent/patient slots.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Centre National de la Recherche Scientifique](https://md.research.elamur.ai/free/en/labs/centre-national-de-la-recherche-scientifique/) — Research institute, Paris, France · 104 works · 60,877 citations
- [Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur](https://md.research.elamur.ai/free/en/labs/laboratoire-d-informatique-pour-la-mecanique-et-les-sciences-de-l-ingenieur/) — Research institute, Orsay, France · 21 works · 7,271 citations
- [Université Paris-Saclay](https://md.research.elamur.ai/free/en/universities/universite-paris-saclay/) — University, Gif-sur-Yvette, France · 34 works · 5,733 citations
- [Université Paris-Sud](https://md.research.elamur.ai/free/en/universities/universite-paris-sud/) — University, Orsay, France · 8 works · 4,961 citations
- [Commissariat à l'Énergie Atomique et aux Énergies Alternatives](https://md.research.elamur.ai/free/en/government/commissariat-a-l-energie-atomique-et-aux-energies-alternatives/) — Government body, Paris, France · 5 works · 350 citations
- [Laboratoire d'Intégration des Systèmes et des Technologies](https://md.research.elamur.ai/free/en/government/laboratoire-d-integration-des-systemes-et-des-technologies/) — Government body, Palaiseau, France · 4 works · 297 citations
- [Hanoi University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hanoi-university-of-science-and-technology/) — University, Hanoi, Vietnam · 2 works · 70 citations
- [Trường Đại học Khoa học và Công nghệ Hà Nội](https://md.research.elamur.ai/free/en/universities/truong-dai-hoc-khoa-hoc-va-cong-nghe-ha-noi/) — University, Hanoi, Vietnam · 2 works · 70 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 109 works

## Cite

Cite as: Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret and Romaric Besançon. 2016. A Dataset for Open Event Extraction in English. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nguyen2016dataset,
  title = {{A Dataset for Open Event Extraction in English}},
  author = {Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret and Romaric Besançon},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/459q7qvijdoi},
  url = {https://doi.org/10.63317/459q7qvijdoi}
}
```

- [← previous Toward incremental dialogue act segmentation in fast-paced interactive dialogue systems](https://md.research.elamur.ai/free/en/papers/manuvinakurike-2016-incremental-dialogue-act-segmentation-fast/)
- [next → An Evaluation of Parser Robustness for Ungrammatical Sentences](https://md.research.elamur.ai/free/en/papers/hashemi-2016-evaluation-parser-robustness-ungrammatical-sentences/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aguilar, J., Beller, C., McNamee, P., Van Durme, B., Strassel, S., Song, Z., and Ellis, J. (2014). [A comparison of the events and relations across ace, ere, tac-kbp, and framenet annotation standards](https://md.research.elamur.ai/free/en/papers/aguilar-2014-comparison-events-relations-across-ace/). In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 45– 53, Baltimore, Maryland, USA, June. Association for Computational Linguistics.
2. Chambers, N. and Jurafsky, D. (2011). Template-Based Information Extraction without the Templates. In 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL 2011), pages 976–986, Portland, Oregon, USA, June. [find](https://scholar.google.com/scholar?q=Chambers%2C+N%2E+and+Jurafsky%2C+D%2E+%282011%29%2E+Template%2DBased+Information+Extraction+without+the+Templates%2E+In+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies+)
3. Chambers, N. (2013). Event Schema Induction with a Probabilistic Entity-Driven Model. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Seattle, Washington, USA, October. [find](https://scholar.google.com/scholar?q=Chambers%2C+N%2E+%282013%29%2E+Event+Schema+Induction+with+a+Probabilistic+Entity%2DDriven+Model%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1797%E2%80%931807%2C+Seattl)
4. Cheung, K. J. C., Poon, H., and Vanderwende, L. (2013). Probabilistic Frame Induction. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 837–846. [↗](https://doi.org/10.48550/arxiv.1302.4813)
5. Doddington, G., Mitchell, A., Przybocki, M., Ramshaw, L., Strassel, S., and Weischedel, R. (2004). [The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation](https://md.research.elamur.ai/free/en/papers/doddington-2004-automatic-content-extraction-ace-program/). In 4 th Conference on Language Resources and Evaluation (LREC 2004), pages 837–840, Lisbon, Portugal.
6. Dongen, S. V. (2000). Graph Clustering by Flow Simulation. Ph.D. thesis, University of Utrecht. [find](https://scholar.google.com/scholar?q=Dongen%2C+S%2E+V%2E+%282000%29%2E+Graph+Clustering+by+Flow+Simulation%2E+Ph%2ED%2E+thesis%2C+University+of+Utrecht%2E)
7. Ellis, J., Getman, J., and Strassel, S. M. (2014). Overview of linguistic resources for the tac kbp 2014 evaluations: Planning, execution, and results. In TAC KBP 2014 Workshop, Gaithersburg, Maryland, USA. National Institute of Standards and Technology. [find](https://scholar.google.com/scholar?q=Ellis%2C+J%2E%2C+Getman%2C+J%2E%2C+and+Strassel%2C+S%2E+M%2E+%282014%29%2E+Overview+of+linguistic+resources+for+the+tac+kbp+2014+evaluations%3A+Planning%2C+execution%2C+and+results%2E+In+TAC+KBP+2014+Workshop%2C+Gaithersburg%2C+Maryland)
8. Grishman, R. and Sundheim, B. (1996). Message Understanding Conference-6: A Brief History. In 16 th International Conference on Computational linguistics (COLING’96), pages 466–471, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Grishman%2C+R%2E+and+Sundheim%2C+B%2E+%281996%29%2E+Message+Understanding+Conference%2D6%3A+A+Brief+History%2E+In+16+th+International+Conference+on+Computational+linguistics+%28COLING%E2%80%9996%29%2C+pages+466%E2%80%93471%2C+Copenhagen%2C+Denmar)
9. Jean-Louis, L., Besançon, R., and Ferret, O. (2011). Text Segmentation and Graph-based Method for Template Filling in Information Extraction. In 5 th International Joint Conference on Natural Language Processing (IJCNLP 2011), pages 723–731, Chiang Mai, Thailand. [find](https://scholar.google.com/scholar?q=Jean%2DLouis%2C+L%2E%2C+Besanc%CC%A7on%2C+R%2E%2C+and+Ferret%2C+O%2E+%282011%29%2E+Text+Segmentation+and+Graph%2Dbased+Method+for+Template+Filling+in+Information+Extraction%2E+In+5+th+International+Joint+Conference+on+Natural+Languag)
10. Kohlschütter, C., Fankhauser, P., and Nejdl, W. (2010). Boilerplate detection using shallow text features. In Third ACM international conference on Web search and data mining (WSDM 2010), pages 441–450. [find](https://scholar.google.com/scholar?q=Kohlschu%CC%88tter%2C+C%2E%2C+Fankhauser%2C+P%2E%2C+and+Nejdl%2C+W%2E+%282010%29%2E+Boilerplate+detection+using+shallow+text+features%2E+In+Third+ACM+international+conference+on+Web+search+and+data+mining+%28WSDM+2010%29%2C+pages+441%E2%80%934)
11. Nguyen, K.-H., Tannier, X., Ferret, O., and Besançon, R. (2015). Generative Event Schema Induction with Entity Disambiguation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing (ACL-IJCNLP 2015), Beijing, China, July. [find](https://scholar.google.com/scholar?q=Nguyen%2C+K%2E%2DH%2E%2C+Tannier%2C+X%2E%2C+Ferret%2C+O%2E%2C+and+Besanc%CC%A7on%2C+R%2E+%282015%29%2E+Generative+Event+Schema+Induction+with+Entity+Disambiguation%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computa)
12. Schank, R. C. and Abelson, R. P. (1977). Scripts, plans, goals and understanding: an inquiry into human knowledge structures. The Artificial intelligence series. L. Erlbaum, Hillsdale, N.J. [find](https://scholar.google.com/scholar?q=Schank%2C+R%2E+C%2E+and+Abelson%2C+R%2E+P%2E+%281977%29%2E+Scripts%2C+plans%2C+goals+and+understanding%3A+an+inquiry+into+human+knowledge+structures%2E+The+Artificial+intelligence+series%2E+L%2E+Erlbaum%2C+Hillsdale%2C+N%2EJ%2E)
13. Song, Z., Bies, A., Strassel, S., Riese, T., Mott, J., Ellis, J., Wright, J., Kulick, S., Ryant, N., and Ma, X. (2015). [From light to rich ere: Annotation of entities, relations, and events](https://md.research.elamur.ai/free/en/papers/song-2015-light-rich-ere-annotation-entities/). In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 89–98, Denver, Colorado, June. Association for Computational Linguistics.
14. Stenetorp, P., Pyysalo, S., Topić, G., Ohta, T., Ananiadou, S., and Tsujii, J. (2012). [BRAT: A Web-based Tool for NLP-assisted Text Annotation](https://md.research.elamur.ai/free/en/papers/stenetorp-2012-brat-web-based-tool-nlp/). In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL’12), pages 102–107, Avignon, France. Association for Computational Linguistics.
15. Theobald, M., Siddharth, J., and Paepcke, A. (2008). Spotsigs: Robust and efficient near duplicate detection in large web collections. In 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’08), pages 563–570, Singapore, Singapore. ACM. [find](https://scholar.google.com/scholar?q=Theobald%2C+M%2E%2C+Siddharth%2C+J%2E%2C+and+Paepcke%2C+A%2E+%282008%29%2E+Spotsigs%3A+Robust+and+efficient+near+duplicate+detection+in+large+web+collections%2E+In+31st+Annual+International+ACM+SIGIR+Conference+on+Research+and)

*Updated 24 October 2026*