---
title: "CogCompTime — paper by Ning et al. (2018)"
description: "CogCompTime is a publicly available system for extracting temporal information from raw English text. 22 references, cited by 32 papers here."
authors: ["Qiang Ning", "Ben Zhou", "Zhili Feng", "Haoruo Peng", "Dan Roth"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations"
source: "https://aclanthology.org/D18-2013/"
doi: "10.18653/v1/d18-2013"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ning-2018-cogcomptime-tool-understanding-time-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ning-2018-cogcomptime-tool-understanding-time-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / CogCompTime: A Tool for Understanding Time in Natural Language

# CogCompTime: A Tool for Understanding Time in Natural Language

**Authors:** [Qiang Ning](https://md.research.elamur.ai/free/en/authors/qiang-ning/), [Ben Zhou](https://md.research.elamur.ai/free/en/authors/ben-zhou/), [Zhili Feng](https://md.research.elamur.ai/free/en/authors/zhili-feng/), [Haoruo Peng](https://md.research.elamur.ai/free/en/authors/haoruo-peng/) and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2018](https://md.research.elamur.ai/free/en/venues/emnlp/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-2013/) · [PDF](https://aclanthology.org/D18-2013.pdf) · [DOI 10.18653/v1/d18-2013](https://doi.org/10.18653/v1/d18-2013) · [OpenAlex](https://openalex.org/W2892202918)

**Citations:** 61 citations (OpenAlex, 2026-10-14) · 32 in our library, of the 149,439 papers whose bibliographies we read

## Summary
CogCompTime is a publicly available system for extracting temporal information from raw English text. It covers two tasks. The first is finding explicit time expressions (Timexes) and converting them to a standard format. The second is recovering the implicit ordering of events through temporal relations (TempRels).

The pipeline works in stages:
- **Preprocessing:** CogCompNLP supplies lemmas, POS tags and SRL.
- **Timexes:** a learned chunker detects them, and hand-written rules normalize them.
- **Events:** a binary classifier detects main-axis events.
- **TempRels:** a perceptron trained with constraint-driven learning labels relations. ILP inference then makes the resulting temporal graph globally consistent.
- **Output:** results appear as a graph view and as a timeline view.

The evaluation uses the TempEval3 corpora, with the TempRel labels swapped for the MATRES annotations, which have higher inter-annotator agreement.

The Timex component performs about as well as leading tools and runs considerably faster. The TempRel scores are well above those reported for TempEval3-era systems, although the authors caution that the datasets differ. The work matters because it bundles recent advances in temporal understanding into one ready-to-use tool. Applications include timeline construction, question answering and causal reasoning.

## Key points
- **Data split:** training uses TimeBank and AQUAINT (256 articles), and testing uses Platinum (20 articles). The TempRel annotations come from MATRES.
- **Timex results on Platinum:** CogCompTime has the best extraction F1 (84.9) and normalization accuracy (84.7) among the compared tools. Its end-to-end F1 (76.8) trails UWTime (82.7) and HeidelTime (78.1). It processed the test set in 7 seconds, versus 18 for HeidelTime, 16 for SUTime and 400 for UWTime.
- **Event extraction:** this is the first event-extraction result reported on MATRES, with F1 85.2, restricted to main-axis verb events. TemporalChunker was built by retraining Illinois-Chunker with BIO tags.
- **TempRel results with gold extraction:** Event-Event F1 is 65.9 (75.0 under a relaxed metric that does not penalize before/after predictions on vague pairs). Event-Timex F1 is 84.6.
- **TempRel results with system extraction:** Event-Event F1 is 52.8 (68.0 relaxed), and Event-Timex F1 is 69.0. For reference, ClearTK scored about 31 under similar conditions on TempEval3. TempRel features include TemProb priors, and inference uses Gurobi.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 221 works · 63,234 citations
- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 184 works · 69,372 citations
- [University of Wisconsin–Madison](https://md.research.elamur.ai/free/en/universities/university-of-wisconsin-madison/) — University, Madison, United States · 42 works · 48,486 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng and Dan Roth. 2018. CogCompTime: A Tool for Understanding Time in Natural Language. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ning2018cogcomptime,
  title = {{CogCompTime: A Tool for Understanding Time in Natural Language}},
  author = {Qiang Ning, Ben Zhou, Zhili Feng, Haoruo Peng and Dan Roth},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year = {2018},
  doi = {10.18653/v1/d18-2013},
  url = {https://doi.org/10.18653/v1/d18-2013}
}
```

- [← previous Shami: A Corpus of Levantine Arabic Dialects](https://md.research.elamur.ai/free/en/papers/kwaik-2018-shami-corpus-levantine-arabic-dialects/)
- [next → A Corpus for Multilingual Document Classification in Eight Languages](https://md.research.elamur.ai/free/en/papers/schwenk-2018-corpus-multilingual-document-classification-eight/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

22 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gabor Angeli, Christopher D. Manning, and Daniel Jurafsky. 2012. Parsing time: Learning to interpret time expressions. In Proc. of the Annual Meeting of the North American Association of Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Gabor+Angeli%2C+Christopher+D%2E+Manning%2C+and+Daniel+Jurafsky%2E+2012%2E+Parsing+time%3A+Learning+to+interpret+time+expressions%2E+In+Proc%2E+of+the+Annual+Meeting+of+the+North+American+Association+of+Computational)
2. Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2018a. Joint reasoning for temporal and causal relations. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 2278–2288, Melbourne, Australia. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Zhili+Feng%2C+Hao+Wu%2C+and+Dan+Roth%2E+2018a%2E+Joint+reasoning+for+temporal+and+causal+relations%2E+In+Proc%2E+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+227)
3. Steven Bethard. 2013. [ClearTK-TimeML: A minimalist approach to TempEval 2013](https://md.research.elamur.ai/free/en/papers/bethard-2013-cleartk-timeml-minimalist-approach-tempeval/). In Second Joint Conference on Lexical and Computational Semantics (* SEM), volume 2, pages 10–14.
4. Qiang Ning, Hao Wu, Haoruo Peng, and Dan Roth. 2018b. Improving temporal relation extraction with a globally acquired statistical resource. In Proc. of the Annual Meeting of the North American Association of Computational Linguistics (NAACL), pages 841–851, New Orleans, Louisiana. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Hao+Wu%2C+Haoruo+Peng%2C+and+Dan+Roth%2E+2018b%2E+Improving+temporal+relation+extraction+with+a+globally+acquired+statistical+resource%2E+In+Proc%2E+of+the+Annual+Meeting+of+the+North+American+Associa)
5. Steven Bethard, Leon Derczynski, Guergana Savova, James Pustejovsky, and Marc Verhagen. 2015. SemEval-2015 Task 6: Clinical TempEval. In Proc. of the 9th International Workshop on SemEval. [find](https://scholar.google.com/scholar?q=Steven+Bethard%2C+Leon+Derczynski%2C+Guergana+Savova%2C+James+Pustejovsky%2C+and+Marc+Verhagen%2E+2015%2E+SemEval%2D2015+Task+6%3A+Clinical+TempEval%2E+In+Proc%2E+of+the+9th+International+Workshop+on+SemEval%2E)
6. Qiang Ning, Hao Wu, and Dan Roth. 2018c. A multi-axis annotation scheme for event temporal relations. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1318– 1328, Melbourne, Australia. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Hao+Wu%2C+and+Dan+Roth%2E+2018c%2E+A+multi%2Daxis+annotation+scheme+for+event+temporal+relations%2E+In+Proc%2E+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+1318%E2%80%93)
7. Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. TACL, 2:273–284. [find](https://scholar.google.com/scholar?q=Nathanael+Chambers%2C+Taylor+Cassidy%2C+Bill+McDowell%2C+and+Steven+Bethard%2E+2014%2E+Dense+event+ordering+with+a+multi%2Dpass+architecture%2E+TACL%2C+2%3A273%E2%80%93284%2E)
8. Haoruo Peng, Yangqiu Song, and Dan Roth. 2016. [Event detection and co-reference with minimal supervision](https://md.research.elamur.ai/free/en/papers/peng-2016-event-detection-co-reference-minimal/). In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
9. Angel X. Chang and Christopher D. Manning. 2012. SUTIME: A library for recognizing and normalizing time expressions. In LREC. [find](https://scholar.google.com/scholar?q=Angel+X%2E+Chang+and+Christopher+D%2E+Manning%2E+2012%2E+SUTIME%3A+A+library+for+recognizing+and+normalizing+time+expressions%2E+In+LREC%2E)
10. Quang Do, Wei Lu, and Dan Roth. 2012. Joint inference for event timeline construction. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Quang+Do%2C+Wei+Lu%2C+and+Dan+Roth%2E+2012%2E+Joint+inference+for+event+timeline+construction%2E+In+Proc%2E+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
11. Vasin Punyakanok and Dan Roth. 2001. The use of classifiers in sequential inference. In Proc. of the Conference on Neural Information Processing Systems (NIPS), pages 995–1001. MIT Press. [find](https://scholar.google.com/scholar?q=Vasin+Punyakanok+and+Dan+Roth%2E+2001%2E+The+use+of+classifiers+in+sequential+inference%2E+In+Proc%2E+of+the+Conference+on+Neural+Information+Processing+Systems+%28NIPS%29%2C+pages+995%E2%80%931001%2E+MIT+Press%2E)
12. Inc. Gurobi Optimization. 2015. Gurobi optimizer reference manual. [find](https://scholar.google.com/scholar?q=Inc%2E+Gurobi+Optimization%2E+2015%2E+Gurobi+optimizer+reference+manual%2E)
13. Jannik Strötgen and Michael Gertz. 2010. HeidelTime: High quality rule-based extraction and normalization of temporal expressions. In Proc. of the 5th International Workshop on SemEval. [find](https://scholar.google.com/scholar?q=Jannik+Stro%CC%88tgen+and+Michael+Gertz%2E+2010%2E+HeidelTime%3A+High+quality+rule%2Dbased+extraction+and+normalization+of+temporal+expressions%2E+In+Proc%2E+of+the+5th+International+Workshop+on+SemEval%2E)
14. Prateek Jindal and Dan Roth. 2013. Extraction of events and temporal expressions from clinical narratives. Journal of Biomedical Informatics (JBI). [find](https://scholar.google.com/scholar?q=Prateek+Jindal+and+Dan+Roth%2E+2013%2E+Extraction+of+events+and+temporal+expressions+from+clinical+narratives%2E+Journal+of+Biomedical+Informatics+%28JBI%29%2E)
15. Naushad UzZaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TEMPEVAL-3: Evaluating time expressions, events, and temporal relations. *SEM, 2:1–9. [find](https://scholar.google.com/scholar?q=Naushad+UzZaman%2C+Hector+Llorens%2C+James+Allen%2C+Leon+Derczynski%2C+Marc+Verhagen%2C+and+James+Pustejovsky%2E+2013%2E+SemEval%2D2013+Task+1%3A+TEMPEVAL%2D3%3A+Evaluating+time+expressions%2C+events%2C+and+temporal+relations%2E)
16. Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, and Vivek Srikumar et al. 2018. CogCompNLP: Your swiss army knife for NLP. In LREC. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Mark+Sammons%2C+Ben+Zhou%2C+Tom+Redman%2C+Christos+Christodoulopoulos%2C+and+Vivek+Srikumar+et+al%2E+2018%2E+CogCompNLP%3A+Your+swiss+army+knife+for+NLP%2E+In+LREC%2E)
17. Kenton Lee, Yoav Artzi, Jesse Dodge, and Luke Zettlemoyer. 2014. Context-dependent semantic parsing for time expressions. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1437–1447. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Yoav+Artzi%2C+Jesse+Dodge%2C+and+Luke+Zettlemoyer%2E+2014%2E+Context%2Ddependent+semantic+parsing+for+time+expressions%2E+In+Proc%2E+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistic)
18. Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky. 2007. SemEval-2007 Task 15: TempEval temporal relation identification. In Proc. of the 4th International Workshop on SemEval, pages 75–80. [find](https://scholar.google.com/scholar?q=Marc+Verhagen%2C+Robert+Gaizauskas%2C+Frank+Schilder%2C+Mark+Hepple%2C+Graham+Katz%2C+and+James+Pustejovsky%2E+2007%2E+SemEval%2D2007+Task+15%3A+TempEval+temporal+relation+identification%2E+In+Proc%2E+of+the+4th+Internatio)
19. Hector Llorens, Nathanael Chambers, Naushad UzZaman, Nasrin Mostafazadeh, James Allen, and James Pustejovsky. 2015. SemEval-2015 Task 5: QA TempEval - evaluating temporal information understanding with question answering. In Proc. of the 9th International Workshop on SemEval. [find](https://scholar.google.com/scholar?q=Hector+Llorens%2C+Nathanael+Chambers%2C+Naushad+UzZaman%2C+Nasrin+Mostafazadeh%2C+James+Allen%2C+and+James+Pustejovsky%2E+2015%2E+SemEval%2D2015+Task+5%3A+QA+TempEval+%2D+evaluating+temporal+information+understanding+wit)
20. Marc Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky. 2010. SemEval-2010 Task 13: TempEval-2. In Proc. of the 5th international workshop on SemEval, pages 57–62. [find](https://scholar.google.com/scholar?q=Marc+Verhagen%2C+Roser+Sauri%2C+Tommaso+Caselli%2C+and+James+Pustejovsky%2E+2010%2E+SemEval%2D2010+Task+13%3A+TempEval%2D2%2E+In+Proc%2E+of+the+5th+international+workshop+on+SemEval%2C+pages+57%E2%80%9362%2E)
21. Ran Zhao, Quang Do, and Dan Roth. 2012. A robust shallow temporal reasoning system. In Proc. of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Ran+Zhao%2C+Quang+Do%2C+and+Dan+Roth%2E+2012%2E+A+robust+shallow+temporal+reasoning+system%2E+In+Proc%2E+of+the+Annual+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NA)
22. Anne-Lyse Minard, Manuela Speranza, Eneko Agirre, Itziar Aldabe, Marieke van Erp, Bernardo Magnini, German Rigau, Ruben Urizar, and Fondazione Bruno Kessler. 2015. SemEval-2015 Task [find](https://scholar.google.com/scholar?q=Anne%2DLyse+Minard%2C+Manuela+Speranza%2C+Eneko+Agirre%2C+Itziar+Aldabe%2C+Marieke+van+Erp%2C+Bernardo+Magnini%2C+German+Rigau%2C+Ruben+Urizar%2C+and+Fondazione+Bruno+Kessler%2E+2015%2E+SemEval%2D2015+Task)

*Updated 29 October 2026*