---
title: "SemEval-2016 Task 12: Clinical TempEval — Bethard 2016"
description: "Clinical TempEval 2016 was a shared task on extracting temporal information from medical records."
authors: ["Steven Bethard", "Guergana Savova", "Wei-Te Chen", "Leon Derczynski", "James Pustejovsky", "Marc Verhagen"]
year: 2016
venue: "Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)"
source: "https://aclanthology.org/S16-1165/"
doi: "10.18653/v1/s16-1165"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bethard-2016-semeval-2016-task-12-clinical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bethard-2016-semeval-2016-task-12-clinical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / SemEval-2016 Task 12: Clinical TempEval

# SemEval-2016 Task 12: Clinical TempEval

**Authors:** [Steven Bethard](https://md.research.elamur.ai/free/en/authors/steven-bethard/), [Guergana Savova](https://md.research.elamur.ai/free/en/authors/guergana-savova/), [Wei-Te Chen](https://md.research.elamur.ai/free/en/authors/wei-te-chen/), [Leon Derczynski](https://md.research.elamur.ai/free/en/authors/leon-derczynski/), [James Pustejovsky](https://md.research.elamur.ai/free/en/authors/james-pustejovsky/), Marc Verhagen · **Year:** 2016 · **Venue:** Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/S16-1165/) · [PDF](https://aclanthology.org/S16-1165.pdf) · [DOI 10.18653/v1/s16-1165](https://doi.org/10.18653/v1/s16-1165) · [OpenAlex](https://openalex.org/W2468432491)

**Citations:** 182 citations (openalex-snapshot · 2026-10-14) · [81](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Clinical TempEval 2016 was a shared task on extracting temporal information from medical records. Systems had to find time expressions, event expressions, their attributes, and temporal links. The data came from the THYME corpus: 600 de-identified clinical and pathology notes about Mayo Clinic cancer patients, annotated with a clinical extension of ISO-TimeML. Nine subtasks were evaluated in two phases. In the first, systems started from raw text. In the second, they received gold events and times and only predicted relations. The organizers simplified the data-access procedure and reused all 2015 data for training, which raised participation from 3 to 14 teams submitting 40 runs. The leading systems, UTHealth and UtahBMI, used structured learners with rich features. On event and time recognition they came within a few hundredths of human agreement. Relation extraction remained harder, but the shortfall on narrative containers was much smaller than in 2015. The task therefore shows real progress and points to relations as the main open problem.

## Key points
- **Corpus:** THYME annotations were distributed as Anafora standoff files in a public repository. The raw note text required a Mayo Clinic data use agreement.
- **Splits:** Train/Dev/Test were divided 50/25/25 and stratified by patient. The splits contain 293/147/151 documents and roughly 3.8k/2.1k/2.0k TIMEX3s.
- **Subtasks:**
  - time spans and classes, including the new PREPOSTEXP class;
  - event spans and attributes (modality, degree, polarity, type);
  - DocTimeRel relations to the document creation time;
  - CONTAINS narrative container relations.
- **Best scores:**
  - CONTAINS F1 was 0.479 (phase 1) and 0.573 (phase 2), both from UTHealth.
  - The best phase-2 DocTimeRel score was 0.843 (UtahBMI), with UTHealth at 0.835.
- **Gap to annotator-adjudicator agreement:**
  - On CONTAINS it was 0.244, down from 0.412 in 2015.
  - On DocTimeRel it was 0.088, slightly up from 0.059. Human agreement itself rose to 0.844 for DocTimeRel and 0.817 for CONTAINS.

## Interesting ideas
- **Normalized attribute score.** Attribute F1 is divided by span F1. This separates "how well are attributes labeled" from "how many mentions were found," which makes systems with very different recall comparable. It suits any pipeline that first detects mentions and then classifies them.
- **Closure-aware relation scoring.** Precision is checked against the transitive closure of the gold relations, and recall against the closure of the system's relations. A system is not penalized for expressing the same timeline through different but logically equivalent links. The approach transfers to graph-structured outputs such as coreference or hierarchy extraction.
- **Two-phase evaluation.** Relations are scored once on raw text and once with gold mentions. This isolates relation-extraction quality from errors that propagate from mention detection, which makes it a useful diagnostic for any end-to-end pipeline.
- **Cheap but strong baselines.**
  - A frequency-based memorization lookup (keep phrases annotated more than half the time, use their most frequent labels).
  - A nearest-event proximity rule for containers.

  Together they quickly show how much a task rewards actual learning beyond surface matching.
- **Reporting two levels of human agreement.** Both annotator-annotator and annotator-adjudicator agreement are given. This yields a more honest ceiling and shows that a "gap to humans" depends on which reference is chosen.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 78 not yet.

- [Visual Representations of Temporal Relations between Events and Time Expressions in News Stories](https://md.research.elamur.ai/free/en/papers/amorim-2025-visual-representations-temporal-relations-between/)
- [A Dataset of Psychiatric Hospital Notes with Temporal Information Annotations](https://md.research.elamur.ai/free/en/papers/miller-2026-dataset-psychiatric-hospital-notes-temporal/)
- [TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions](https://md.research.elamur.ai/free/en/papers/ning-2020-torque-reading-comprehension-dataset-temporal/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bethard2016semeval,
  title = {{SemEval-2016 Task 12: Clinical TempEval}},
  author = {Bethard, Steven and Savova, Guergana and Chen, Wei-Te and Derczynski, Leon and Pustejovsky, James and Verhagen, Marc},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1165},
  url = {https://doi.org/10.18653/v1/s16-1165}
}
```

- [← previous SemEval-2016 Task 10: Detecting Minimal Semantic Units and their Meanings (DiMSUM)](https://md.research.elamur.ai/free/en/papers/schneider-2016-semeval-2016-task-10-detecting/)
- [next → SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdulrahman Khalifa AAl Abdulsalam, Sumithra Velupillai, and Stephane Meystre. 2016. UtahBMI at SemEval-2016 Task 12: Extracting temporal information from clinical text. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1256–1262, San Diego, California, J… [find](https://scholar.google.com/scholar?q=Abdulrahman+Khalifa+AAl+Abdulsalam%2C+Sumithra+Velupillai%2C+and+Stephane+Meystre%2E+2016%2E+UtahBMI+at+SemEval%2D2016+Task+12%3A+Extracting+temporal+information+from+clinical+text%2E+In+Proceedings+of+the+10th+Int)
2. Steven Bethard, Leon Derczynski, Guergana Savova, James Pustejovsky, and Marc Verhagen. 2015. Semeval-2015 task 6: Clinical tempeval. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 806–814, Denver, Colorado, June. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Steven+Bethard%2C+Leon+Derczynski%2C+Guergana+Savova%2C+James+Pustejovsky%2C+and+Marc+Verhagen%2E+2015%2E+Semeval%2D2015+task+6%3A+Clinical+tempeval%2E+In+Proceedings+of+the+9th+International+Workshop+on+Semantic+Evalu)
3. Tommaso Caselli and Roser Morante. 2016. VUACLTL at SemEval-2016 Task 12: A crf pipeline to clinical temporal. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, California, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tommaso+Caselli+and+Roser+Morante%2E+2016%2E+VUACLTL+at+SemEval%2D2016+Task+12%3A+A+crf+pipeline+to+clinical+temporal%2E+In+Proceedings+of+the+10th+International+Workshop+on+Semantic+Evaluation+%28SemEval+2016%29%2C+)
4. Wei-Te Chen and Will Styler. 2013. Anafora: A web-based general purpose annotation tool. In Proceedings of the 2013 NAACL HLT Demonstration Session, pages 14–19, Atlanta, Georgia, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wei%2DTe+Chen+and+Will+Styler%2E+2013%2E+Anafora%3A+A+web%2Dbased+general+purpose+annotation+tool%2E+In+Proceedings+of+the+2013+NAACL+HLT+Demonstration+Session%2C+pages+14%E2%80%9319%2C+Atlanta%2C+Georgia%2C+June%2E+Association+fo)
5. Veera Raghavendra Chikka. 2016. CDE-IIITH at SemEval-2016 Task 12: Extraction of temporal information from clinical documents using machine learning techniques. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, California, June. Association for Compu… [find](https://scholar.google.com/scholar?q=Veera+Raghavendra+Chikka%2E+2016%2E+CDE%2DIIITH+at+SemEval%2D2016+Task+12%3A+Extraction+of+temporal+information+from+clinical+documents+using+machine+learning+techniques%2E+In+Proceedings+of+the+10th+Internationa)
6. Arman Cohan, Kevin Meurer, and Nazli Goharian. 2016. GUIR at SemEval-2016 Task 12: Temporal information extraction from clinical narratives. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, California, June. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Arman+Cohan%2C+Kevin+Meurer%2C+and+Nazli+Goharian%2E+2016%2E+GUIR+at+SemEval%2D2016+Task+12%3A+Temporal+information+extraction+from+clinical+narratives%2E+In+Proceedings+of+the+10th+International+Workshop+on+Semant)
7. Jason Fries. 2016. Brundlefly at SemEval-2016 Task 12: Recurrent neural networks vs. joint inference for clinical temporal information extraction. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, California, June. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Jason+Fries%2E+2016%2E+Brundlefly+at+SemEval%2D2016+Task+12%3A+Recurrent+neural+networks+vs%2E+joint+inference+for+clinical+temporal+information+extraction%2E+In+Proceedings+of+the+10th+International+Workshop+on+)
8. Hee-Jin Lee, Hua Xu, Jingqi Wang, Yaoyun Zhang, Sungrim Moon, Jun Xu, and Yonghui Wu. 2016. UTHealth at SemEval-2016 Task 12: Temporal information extraction from clinical notes - uthealth’s system for the 2016 clinical tempeval challenge. In Proceedings of the 10th International Workshop on Semanti… [find](https://scholar.google.com/scholar?q=Hee%2DJin+Lee%2C+Hua+Xu%2C+Jingqi+Wang%2C+Yaoyun+Zhang%2C+Sungrim+Moon%2C+Jun+Xu%2C+and+Yonghui+Wu%2E+2016%2E+UTHealth+at+SemEval%2D2016+Task+12%3A+Temporal+information+extraction+from+clinical+notes+%2D+uthealth%E2%80%99s+system+fo)
9. Artuur Leeuwenberg and Marie-Francine Moens. 2016. KULeuven-LIIR at SemEval-2016 Task 12: Detecting narrative containment in clinical records. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, California, June. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Artuur+Leeuwenberg+and+Marie%2DFrancine+Moens%2E+2016%2E+KULeuven%2DLIIR+at+SemEval%2D2016+Task+12%3A+Detecting+narrative+containment+in+clinical+records%2E+In+Proceedings+of+the+10th+International+Workshop+on+Sema)
10. Peng Li and Heng Huang. 2016. UTA MLNLP at SemEval-2016 Task 12. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, California, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peng+Li+and+Heng+Huang%2E+2016%2E+UTA+MLNLP+at+SemEval%2D2016+Task+12%2E+In+Proceedings+of+the+10th+International+Workshop+on+Semantic+Evaluation+%28SemEval+2016%29%2C+San+Diego%2C+California%2C+June%2E+Association+for+C)
11. Chen Lin, Dmitriy Dligach, Timothy A Miller, Steven Bethard, and Guergana K Savova. 2015. Multilayered temporal modeling for the clinical domain. Journal of the American Medical Informatics Association. [find](https://scholar.google.com/scholar?q=Chen+Lin%2C+Dmitriy+Dligach%2C+Timothy+A+Miller%2C+Steven+Bethard%2C+and+Guergana+K+Savova%2E+2015%2E+Multilayered+temporal+modeling+for+the+clinical+domain%2E+Journal+of+the+American+Medical+Informatics+Associatio)
12. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Ma… [find](https://scholar.google.com/scholar?q=Christopher+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+Bethard%2C+and+David+McClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Proceedings+of+52nd+Annual+Meeting) [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
13. James Pustejovsky and Amber Stubbs. 2011. Increasing informativeness in temporal annotation. In Proceedings of the 5th Linguistic Annotation Workshop, pages 152–160, Portland, Oregon, USA, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=James+Pustejovsky+and+Amber+Stubbs%2E+2011%2E+Increasing+informativeness+in+temporal+annotation%2E+In+Proceedings+of+the+5th+Linguistic+Annotation+Workshop%2C+pages+152%E2%80%93160%2C+Portland%2C+Oregon%2C+USA%2C+June%2E+Assoc) [our card](https://md.research.elamur.ai/free/en/papers/pustejovsky-2011-increasing-informativeness-temporal-annotation/)
14. Sarath P R, Manikandan R, and Yoshiki Niwa. 2016. Hitachi at SemEval-2016 Task 12: Extraction of temporal information for the 2016 clinical tempeval challenge. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), San Diego, California, June. Association for Comput… [find](https://scholar.google.com/scholar?q=Sarath+P+R%2C+Manikandan+R%2C+and+Yoshiki+Niwa%2E+2016%2E+Hitachi+at+SemEval%2D2016+Task+12%3A+Extraction+of+temporal+information+for+the+2016+clinical+tempeval+challenge%2E+In+Proceedings+of+the+10th+International)
15. William F. Styler, IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. 2014a. Temporal annotation in the clinical domain. Transactions of the Association for Computational Linguistics, 2:143–15… [find](https://scholar.google.com/scholar?q=William+F%2E+Styler%2C+IV%2C+Steven+Bethard%2C+Sean+Finan%2C+Martha+Palmer%2C+Sameer+Pradhan%2C+Piet+de+Groen%2C+Brad+Erickson%2C+Timothy+Miller%2C+Chen+Lin%2C+Guergana+Savova%2C+and+James+Pustejovsky%2E+2014a%2E+Temporal+annota) [our card](https://md.research.elamur.ai/free/en/papers/iv-2014-temporal-annotation-clinical-domain/)
16. William F. Styler, IV, Guergana Savova, Martha Palmer, James Pustejovsky, Tim O’Gorman, and Piet C. de Groen. 2014b. THYME annotation guidelines, 2. [find](https://scholar.google.com/scholar?q=William+F%2E+Styler%2C+IV%2C+Guergana+Savova%2C+Martha+Palmer%2C+James+Pustejovsky%2C+Tim+O%E2%80%99Gorman%2C+and+Piet+C%2E+de+Groen%2E+2014b%2E+THYME+annotation+guidelines%2C+2%2E)
17. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013. Evaluating temporal relations in clinical text: 2012 i2b2 challenge. Journal of the American Medical Informatics Association, 20(5):806–813. Julien Tourille, Olivier Ferret, Aurélie Névéol, and Xavier Tannier. 2016. LIMSI-COT at SemEval-2016 Task… [find](https://scholar.google.com/scholar?q=Weiyi+Sun%2C+Anna+Rumshisky%2C+and+Ozlem+Uzuner%2E+2013%2E+Evaluating+temporal+relations+in+clinical+text%3A+2012+i2b2+challenge%2E+Journal+of+the+American+Medical+Informatics+Association%2C+20%285%29%3A806%E2%80%93813%2E+Julien+T)
18. Naushad UzZaman and James Allen. 2011. Temporal evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 351–356, Portland, Oregon, USA, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Naushad+UzZaman+and+James+Allen%2E+2011%2E+Temporal+evaluation%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies%2C+pages+351%E2%80%93356%2C+Port)
19. Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of… [find](https://scholar.google.com/scholar?q=Naushad+UzZaman%2C+Hector+Llorens%2C+Leon+Derczynski%2C+James+Allen%2C+Marc+Verhagen%2C+and+James+Pustejovsky%2E+2013%2E+SemEval%2D2013+Task+1%3A+TempEval%2D3%3A+Evaluating+Time+Expressions%2C+Events%2C+and+Temporal+Relations%2E)
20. Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky. 2007. SemEval-2007 Task 15: TempEval Temporal Relation Identification. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 75–80, Prague, Czech Republic, J… [find](https://scholar.google.com/scholar?q=Marc+Verhagen%2C+Robert+Gaizauskas%2C+Frank+Schilder%2C+Mark+Hepple%2C+Graham+Katz%2C+and+James+Pustejovsky%2E+2007%2E+SemEval%2D2007+Task+15%3A+TempEval+Temporal+Relation+Identification%2E+In+Proceedings+of+the+Fourth+I)
21. Marc Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky. 2010. SemEval-2010 Task 13: TempEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 57–62, Uppsala, Sweden, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marc+Verhagen%2C+Roser+Sauri%2C+Tommaso+Caselli%2C+and+James+Pustejovsky%2E+2010%2E+SemEval%2D2010+Task+13%3A+TempEval%2D2%2E+In+Proceedings+of+the+5th+International+Workshop+on+Semantic+Evaluation%2C+pages+57%E2%80%9362%2C+Uppsal)