--- title: "Increasing Informativeness in Temporal — Pustejovsky 2011" description: "Under TimeML, linking every pair of events and times in a document is infeasible, so the guideline decides which TLINKs annotators produce." authors: ["James Pustejovsky", "Amber Stubbs"] year: 2011 venue: "Proceedings of the 5th Linguistic Annotation Workshop" source: "https://aclanthology.org/W11-0419/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/pustejovsky-2011-increasing-informativeness-temporal-annotation/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/pustejovsky-2011-increasing-informativeness-temporal-annotation/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2011](https://md.research.elamur.ai/free/papers/2011/) / Increasing Informativeness in Temporal Annotation # Increasing Informativeness in Temporal Annotation **Authors:** [James Pustejovsky](https://md.research.elamur.ai/free/authors/james-pustejovsky/), Amber Stubbs · **Year:** 2011 · **Venue:** Proceedings of the 5th Linguistic Annotation Workshop · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W11-0419/) · [PDF](https://aclanthology.org/W11-0419.pdf) · [OpenAlex](https://openalex.org/W1913805045) **Citations:** 64 citations (openalex-snapshot · 2026-10-14) · [43](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Under TimeML, linking every pair of events and times in a document is infeasible, so the guideline decides which TLINKs annotators produce. An error analysis of TimeBank TLINKs showed that the guideline leaves out useful link types and pushes annotators toward weak ones, mostly orderings of events against the Document Creation Time (DCT). The authors propose the narrative container (NC): a default, genre-dependent interval around publication time in which events with no explicit temporal anchor are assumed to occur. They also define Narrative Time (the currently active anchor, which shifts as new timexes appear) and Narrative Scope (the span from the earliest to the latest event mentioned). Three annotators re-annotated TimeBank, anchoring each event either to the closest time expression or to the NC with a reduced relation set. Agreement on anchors was fairly high. The new links placed events inside bounded intervals much more often than the original TimeBank links did. The work shows that guideline design, not only the specification, determines how informative a temporal corpus is. ## Key points - The suggested default NC length depends on publication type: roughly 24 hours (one business day) for print newspapers, 2–10 hours for newswire, and a week or more for weekly or monthly outlets. - The re-annotation used only three relations (before, after, is included). Event-event links were excluded. - About 50% of events were linked to the NC with "is included." Under the original TimeBank annotation, such events would typically get only a vaguer before/overlap relation to the DCT. - Fleiss' kappa for temporal anchors averaged 0.74 (range −0.04 to 1), computed in R with the irr package. Negative values occur for articles where annotators linked everything to the NC, which is an artifact of low variability rather than real disagreement. - In the case study wsj_1042.tml (NC covering the day preceding the DCT up to and including the DCT itself), every event received at least one temporal constraint, and most received both a left and a right bound. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/) ## Cited by Cited by 43 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 39 not yet. - [Cross-document coreference: An approach to capturing coreference without context](https://md.research.elamur.ai/free/papers/bettner-2019-cross-document-coreference-approach-capturing/) - [Richer Event Description: Integrating event coreference with temporal, causal and bridging annotation](https://md.research.elamur.ai/free/papers/gorman-2016-richer-event-description-integrating-event/) - [A Dataset of Psychiatric Hospital Notes with Temporal Information Annotations](https://md.research.elamur.ai/free/papers/miller-2026-dataset-psychiatric-hospital-notes-temporal/) - [Opinion Piece: Can we Fix the Scope for Coreference? Problems and Solutions for Benchmarks beyond OntoNotes](https://md.research.elamur.ai/free/papers/zeldes-2022-opinion-piece-can-we-fix/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{pustejovsky2011increasing, title = {{Increasing Informativeness in Temporal Annotation}}, author = {Pustejovsky, James and Stubbs, Amber}, booktitle = {Proceedings of the 5th Linguistic Annotation Workshop}, year = {2011}, url = {https://aclanthology.org/W11-0419/} } ``` - [← previous Improving Arabic Dependency Parsing with Form-based and Functional Morphological Features](https://md.research.elamur.ai/free/papers/aclbulk-2011-marton-improving-arabic-dependency-parsing-form/) - [next → Inference Protocols for Coreference Resolution](https://md.research.elamur.ai/free/papers/chang-2011-inference-protocols-coreference-resolution/) - [↑ back to papers of 2011](https://md.research.elamur.ai/free/papers/2011/) ## References 16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. James Allen. 1984. Towards a general theory of action and time. Arificial Intelligence, 23:123–154. [find](https://scholar.google.com/scholar?q=James+Allen%2E+1984%2E+Towards+a+general+theory+of+action+and+time%2E+Arificial+Intelligence%2C+23%3A123%E2%80%93154%2E) 2. Douglas Biber. 2009. Register, Genre, and Style. [find](https://scholar.google.com/scholar?q=Douglas+Biber%2E+2009%2E+Register%2C+Genre%2C+and+Style%2E) 3. J. L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. [find](https://scholar.google.com/scholar?q=J%2E+L%2E+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+Bulletin%2C+76%285%29%3A378%E2%80%93382%2E) 4. Matthias Gamer, Jim Lemon, and Ian Fellows Pus-pendra Singh ¡puspendra.pusp22@gmail.com¿, 2010. irr: Various Coefficients of Interrater Reliability and Agreement. R package version 0.83. [find](https://scholar.google.com/scholar?q=Matthias+Gamer%2C+Jim+Lemon%2C+and+Ian+Fellows+Pus%2Dpendra+Singh+%C2%A1puspendra%2Epusp22%40gmail%2Ecom%C2%BF%2C+2010%2E+irr%3A+Various+Coefficients+of+Interrater+Reliability+and+Agreement%2E+R+package+version+0%2E83%2E) 5. Nancy Ide and Harry Bunt. 2010. Anatomy of annotation schemes: Mappings to graf. In In Proceedings 4th Linguistic Annotation Workshop (LAW IV). [find](https://scholar.google.com/scholar?q=Nancy+Ide+and+Harry+Bunt%2E+2010%2E+Anatomy+of+annotation+schemes%3A+Mappings+to+graf%2E+In+In+Proceedings+4th+Linguistic+Annotation+Workshop+%28LAW+IV%29%2E) 6. Nancy Ide. 2007. Annotation science: From theory to practice and use: Data structures for linguistics resources and applications. In In Proceedings of the Bi-enniel GLDV Conference. [find](https://scholar.google.com/scholar?q=Nancy+Ide%2E+2007%2E+Annotation+science%3A+From+theory+to+practice+and+use%3A+Data+structures+for+linguistics+resources+and+applications%2E+In+In+Proceedings+of+the+Bi%2Denniel+GLDV+Conference%2E) 7. David Lee. 2001. Genres, registers, text types, domains, and styles: Clarifying the concepts and navigating a path through the bnc jungle. Language Learning & Technology, 5(3.3):37–72. [find](https://scholar.google.com/scholar?q=David+Lee%2E+2001%2E+Genres%2C+registers%2C+text+types%2C+domains%2C+and+styles%3A+Clarifying+the+concepts+and+navigating+a+path+through+the+bnc+jungle%2E+Language+Learning+%26+Technology%2C+5%283%2E3%29%3A37%E2%80%9372%2E) 8. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The penn discourse treebank. In In Proceedings of LREC 2004. [find](https://scholar.google.com/scholar?q=Eleni+Miltsakaki%2C+Rashmi+Prasad%2C+Aravind+Joshi%2C+and+Bonnie+Webber%2E+2004%2E+The+penn+discourse+treebank%2E+In+In+Proceedings+of+LREC+2004%2E) [our card](https://md.research.elamur.ai/free/papers/miltsakaki-2004-penn-discourse-treebank/) 9. Massimo Poesio. 2004. Discourse annotation and semantic annotation in the gnome corpus. In In Proceedings of the ACL Workshop on Discourse Annotation. [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2E+2004%2E+Discourse+annotation+and+semantic+annotation+in+the+gnome+corpus%2E+In+In+Proceedings+of+the+ACL+Workshop+on+Discourse+Annotation%2E) 10. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). [find](https://scholar.google.com/scholar?q=Rashmi+Prasad%2C+Nikhil+Dinesh%2C+Alan+Lee%2C+Eleni+Miltsakaki%2C+Livio+Robaldo%2C+Aravind+Joshi%2C+and+Bonnie+Webber%2E+2008%2E+The+penn+discourse+treebank+2%2E0%2E+In+In+Proceedings+of+the+6th+International+Conference+) [our card](https://md.research.elamur.ai/free/papers/disc-2008-prasad-penn-discourse-treebank-2-0/) 11. James Pustejovsky, Patrick Hanks, Roser Saurı̀, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, and Marcia Lazo. 2003. The timebank corpus. In Dawn Archer, Paul Rayson, Andrew Wilson, and Tony McEnery, editors, Proceedings of the Corpus Linguistics… [find](https://scholar.google.com/scholar?q=James+Pustejovsky%2C+Patrick+Hanks%2C+Roser+Saur%C4%B1%CC%80%2C+Andrew+See%2C+Robert+Gaizauskas%2C+Andrea+Setzer%2C+Dragomir+Radev%2C+Beth+Sundheim%2C+David+Day%2C+Lisa+Ferro%2C+and+Marcia+Lazo%2E+2003%2E+The+timebank+corpus%2E+In+Dawn+) 12. James Pustejovsky, Robert Knippen, Jessica Littman, and Roser Saurı́. 2005. Temporal and event information in natural language text. Language Resources and Evaluation, 39:123–164, May. [find](https://scholar.google.com/scholar?q=James+Pustejovsky%2C+Robert+Knippen%2C+Jessica+Littman%2C+and+Roser+Saur%C4%B1%CC%81%2E+2005%2E+Temporal+and+event+information+in+natural+language+text%2E+Language+Resources+and+Evaluation%2C+39%3A123%E2%80%93164%2C+May%2E) 13. R Team, 2009. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. [find](https://scholar.google.com/scholar?q=R+Team%2C+2009%2E+R%3A+A+Language+and+Environment+for+Statistical+Computing%2E+R+Foundation+for+Statistical+Computing%2C+Vienna%2C+Austria%2E+ISBN+3%2D900051%2D07%2D0%2E) 14. Roser Saurı́, Jessica Littman, Bob Knippen, Robert Gaizauskas, Andrea Setzer, and James Pustejovsky, 2006. TimeML Annotation Guidelines, version 1.2.1 edition, January. [find](https://scholar.google.com/scholar?q=Roser+Saur%C4%B1%CC%81%2C+Jessica+Littman%2C+Bob+Knippen%2C+Robert+Gaizauskas%2C+Andrea+Setzer%2C+and+James+Pustejovsky%2C+2006%2E+TimeML+Annotation+Guidelines%2C+version+1%2E2%2E1+edition%2C+January%2E) 15. Carlota Smith. 2003. Modes of Discourse. Cambridge University Press, Cambridge, UK. [find](https://scholar.google.com/scholar?q=Carlota+Smith%2E+2003%2E+Modes+of+Discourse%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2E) 16. Marc Vilain, Henry Kautz, and Peter Beek. 1986. Constraint propagation algorithms for temporal reasoning. In Readings in Qualitative Reasoning about Physical Systems, pages 377–382. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Marc+Vilain%2C+Henry+Kautz%2C+and+Peter+Beek%2E+1986%2E+Constraint+propagation+algorithms+for+temporal+reasoning%2E+In+Readings+in+Qualitative+Reasoning+about+Physical+Systems%2C+pages+377%E2%80%93382%2E+Morgan+Kaufmann%2E)