---
title: "Publicly Available Clinical BERT Embeddings — Alsentzer 2019"
description: "Contextual encoders like BERT had transformed general-domain NLP, but no pretrained BERT for clinical narratives (e.g., physician notes) was publicly…"
authors: ["Emily Alsentzer", "John R. Murphy", "Willie Boag", "Wei-Hung Weng", "Di Jin", "Tristan Naumann", "Matthew B. A. McDermott"]
year: 2019
venue: "arXiv:1904.03323"
source: "https://arxiv.org/abs/1904.03323"
doi: "10.48550/arxiv.1904.03323"
arxiv: "1904.03323"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alsentzer-2019-publicly-available-clinical-bert-embeddings/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alsentzer-2019-publicly-available-clinical-bert-embeddings/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Publicly Available Clinical BERT Embeddings

# Publicly Available Clinical BERT Embeddings

**Authors:** Emily Alsentzer, John R. Murphy, Willie Boag, [Wei-Hung Weng](https://md.research.elamur.ai/free/en/authors/wei-hung-weng/), [Di Jin](https://md.research.elamur.ai/free/en/authors/di-jin/), [Tristan Naumann](https://md.research.elamur.ai/free/en/authors/tristan-naumann/), Matthew B. A. McDermott · **Year:** 2019 · **Venue:** arXiv:1904.03323 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1904.03323) · [PDF](https://arxiv.org/pdf/1904.03323) · [DOI 10.48550/arxiv.1904.03323](https://doi.org/10.48550/arxiv.1904.03323) · [arXiv 1904.03323](https://arxiv.org/abs/1904.03323) · [OpenAlex](https://openalex.org/W2925863688)

**Citations:** 693 citations (openalex-snapshot · 2026-10-14) · [512](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Contextual encoders like BERT had transformed general-domain NLP, but no pretrained BERT for clinical narratives (e.g., physician notes) was publicly available. Such text differs noticeably from both web prose and biomedical literature. The authors pretrained BERT variants on roughly 2 million notes from MIMIC-III v1.4. They built one model from all note types and one from discharge summaries alone. Each was initialized either from BERT-Base or from BioBERT. Training followed the standard recipe, and evaluation used only a single linear layer on top of the fine-tuned encoder. Across MedNLI and four i2b2 tasks, the clinical models beat general BERT and BioBERT on MedNLI, i2b2 2010 and i2b2 2012. MedNLI accuracy reached a new state of the art. On the two de-identification tasks, the clinical models showed no gain. The authors attribute this to a distribution mismatch: MIMIC replaces PHI with placeholder tokens, whereas the de-ID corpora insert realistic synthetic surrogates. The main value of the work is the free release of the models, which spares other labs weeks of GPU time.

## Key points
- Four released checkpoints are available at github.com/EmilyAlsentzer/clinicalBERT: Clinical BERT, Discharge Summary BERT, Bio+Clinical BERT and Bio+Discharge Summary BERT.
- On MedNLI, the Bio+Clinical variants scored 82.7% accuracy. This compares with 77.6% for BERT, 80.8% for BioBERT and the earlier best of 73.5% from InferSent.
- On i2b2 2010 concept extraction, Bio+Discharge Summary BERT reached an exact F1 of 87.8. BioBERT scored 86.5 and general BERT 83.5.
- On the de-ID tasks (i2b2 2006, i2b2 2014), general BERT or BioBERT matched or outperformed the clinical models. For example, on i2b2 2006, Clinical BERT scored 91.5 against 94.8 for BioBERT.
- Pretraining took about 17–18 days on a single GTX TITAN X GPU.
- Stated limitations:
  - The task heads are simple.
  - The notes come only from the ICU of one hospital (BIDMC).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

512 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 504 not yet.

- [SciBERT: A Pretrained Language Model for Scientific Text](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://md.research.elamur.ai/free/en/papers/hedderich-2021-survey-recent-approaches-natural-language/)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://md.research.elamur.ai/free/en/papers/huang-2019-clinicalbert-modeling-clinical-notes-predicting/)
- [Towards Debiasing Sentence Representations](https://md.research.elamur.ai/free/en/papers/liang-2020-debiasing-sentence-representations/)
- [Coreference Resolution for the Biomedical Domain: A Survey](https://md.research.elamur.ai/free/en/papers/lu-2021-coreference-resolution-biomedical-domain-survey/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alsentzer2019publicly,
  title = {{Publicly Available Clinical BERT Embeddings}},
  author = {Alsentzer, Emily and Murphy, John R. and Boag, Willie and Weng, Wei-Hung and Jin, Di and Naumann, Tristan and McDermott, Matthew B. A.},
  year = {2019},
  doi = {10.48550/arxiv.1904.03323},
  eprint = {1904.03323},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.03323}
}
```

- [← previous Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019)](https://md.research.elamur.ai/free/en/papers/anon-2019-proceedings-third-workshop-universal-dependencies/)
- [next → PubMedQA: A Dataset for Biomedical Research Question Answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

22 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E)
2. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. arXiv:1705.02364 [cs]. ArXiv: 1705.02364. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Loic+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+Learning+of+Universal+Sentence+Representations+from+Natural+Language+Inference+Data%2E+arXiv%3A1705%2E02364+%5B) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. ArXiv: 1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+arXiv%3A1810%2E04805+%5Bcs%5D%2E+ArXiv%3A+1810%2E04805%2E)
4. Noemie Elhadad, Sameer Pradhan, Sharon Gorman, Suresh Manandhar, Wendy Chapman, and Guergana Savova. SemEval-2015 Task 14: Analysis of Clinical Text. page 8. [find](https://scholar.google.com/scholar?q=Noemie+Elhadad%2C+Sameer+Pradhan%2C+Sharon+Gorman%2C+Suresh+Manandhar%2C+Wendy+Chapman%2C+and+Guergana+Savova%2E+SemEval%2D2015+Task+14%3A+Analysis+of+Clinical+Text%2E+page+8%2E)
5. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. arXiv:1801.06146 [cs, stat]. ArXiv: 1801.06146. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+Language+Model+Fine%2Dtuning+for+Text+Classification%2E+arXiv%3A1801%2E06146+%5Bcs%2C+stat%5D%2E+ArXiv%3A+1801%2E06146%2E)
6. Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data, 3:160035. [find](https://scholar.google.com/scholar?q=Alistair+E%2E+W%2E+Johnson%2C+Tom+J%2E+Pollard%2C+Lu+Shen%2C+Liwei+H%2E+Lehman%2C+Mengling+Feng%2C+Mohammad+Ghassemi%2C+Benjamin+Moody%2C+Peter+Szolovits%2C+Leo+Anthony+Celi%2C+and+Roger+G%2E+Mark%2E+2016%2E+MIMIC%2DIII%2C+a+freely+acce) [our card](https://md.research.elamur.ai/free/en/papers/johnson-2016-mimic-iii-freely-accessible-critical/)
7. Kaung Khin, Philipp Burckhardt, and Rema Padman. 2018. A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation. arXiv:1810.01570 [cs]. ArXiv: 1810.01570. — [arXiv:1810.01570](https://arxiv.org/abs/1810.01570) [find](https://scholar.google.com/scholar?q=Kaung+Khin%2C+Philipp+Burckhardt%2C+and+Rema+Padman%2E+2018%2E+A+Deep+Learning+Architecture+for+De%2Didentification+of+Patient+Notes%3A+Implementation+and+Evaluation%2E+arXiv%3A1810%2E01570+%5Bcs%5D%2E+ArXiv%3A+1810%2E01570%2E+%E2%80%94+%5B)
8. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. arXiv:1901.08746 [cs]. ArXiv: 1901.08746. [find](https://scholar.google.com/scholar?q=Jinhyuk+Lee%2C+Wonjin+Yoon%2C+Sungdong+Kim%2C+Donghyeon+Kim%2C+Sunkyu+Kim%2C+Chan+Ho+So%2C+and+Jaewoo+Kang%2E+2019%2E+BioBERT%3A+a+pre%2Dtrained+biomedical+language+representation+model+for+biomedical+text+mining%2E+arXiv%3A) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
9. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing+)
10. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. arXiv:1902.07669 [cs]. ArXiv: 1902.07669. [find](https://scholar.google.com/scholar?q=Mark+Neumann%2C+Daniel+King%2C+Iz+Beltagy%2C+and+Waleed+Ammar%2E+2019%2E+ScispaCy%3A+Fast+and+Robust+Models+for+Biomedical+Natural+Language+Processing%2E+arXiv%3A1902%2E07669+%5Bcs%5D%2E+ArXiv%3A+1902%2E07669%2E)
11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+process)
12. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv:1802.05365 [cs]. ArXiv: 1802.05365. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+arXiv%3A1802%2E05365+%5Bcs%5D%2E+ArXiv%3A+1802%2E0536)
13. Sameer Pradhan, Nomie Elhadad, Wendy Chapman, Suresh Manandhar, and Guergana Savova. 2014. SemEval-2014 Task 7: Analysis of Clinical Text. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 54–62, Dublin, Ireland. Association for Computational Linguistics a… [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Nomie+Elhadad%2C+Wendy+Chapman%2C+Suresh+Manandhar%2C+and+Guergana+Savova%2E+2014%2E+SemEval%2D2014+Task+7%3A+Analysis+of+Clinical+Text%2E+In+Proceedings+of+the+8th+International+Workshop+on+Semantic+)
14. Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text Chunking using Transformation-Based Learning. arXiv:cmp-lg/9505040. ArXiv: cmp-lg/9505040. [find](https://scholar.google.com/scholar?q=Lance+A%2E+Ramshaw+and+Mitchell+P%2E+Marcus%2E+1995%2E+Text+Chunking+using+Transformation%2DBased+Learning%2E+arXiv%3Acmp%2Dlg%2F9505040%2E+ArXiv%3A+cmp%2Dlg%2F9505040%2E)
15. Alexey Romanov and Chaitanya Shivade. 2018. Lessons from Natural Language Inference in the Clinical Domain. arXiv:1808.06752 [cs]. ArXiv: 1808.06752. [find](https://scholar.google.com/scholar?q=Alexey+Romanov+and+Chaitanya+Shivade%2E+2018%2E+Lessons+from+Natural+Language+Inference+in+the+Clinical+Domain%2E+arXiv%3A1808%2E06752+%5Bcs%5D%2E+ArXiv%3A+1808%2E06752%2E) [our card](https://md.research.elamur.ai/free/en/papers/romanov-2018-lessons-natural-language-inference-clinical/)
16. Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts. 2019. Enhancing Clinical Concept Extraction with Contextual Embedding. arXiv:1902.08691 [cs]. ArXiv: 1902.08691. — [arXiv:1902.08691](https://arxiv.org/abs/1902.08691) [find](https://scholar.google.com/scholar?q=Yuqi+Si%2C+Jingqi+Wang%2C+Hua+Xu%2C+and+Kirk+Roberts%2E+2019%2E+Enhancing+Clinical+Concept+Extraction+with+Contextual+Embedding%2E+arXiv%3A1902%2E08691+%5Bcs%5D%2E+ArXiv%3A+1902%2E08691%2E+%E2%80%94+%5BarXiv%3A1902%2E08691%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
17. Amber Stubbs, Christopher Kotfila, and zlem Uzuner. 2015. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1. Journal of Biomedical Informatics, 58 Suppl:S11–19. [find](https://scholar.google.com/scholar?q=Amber+Stubbs%2C+Christopher+Kotfila%2C+and+zlem+Uzuner%2E+2015%2E+Automated+systems+for+the+de%2Didentification+of+longitudinal+clinical+narratives%3A+Overview+of+2014+i2b2%2FUTHealth+shared+task+Track+1%2E+Journal+o)
18. Amber Stubbs and zlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus. Journal of Biomedical Informatics, 58 Suppl:S20–29. [find](https://scholar.google.com/scholar?q=Amber+Stubbs+and+zlem+Uzuner%2E+2015%2E+Annotating+longitudinal+clinical+narratives+for+de%2Didentification%3A+The+2014+i2b2%2FUTHealth+corpus%2E+Journal+of+Biomedical+Informatics%2C+58+Suppl%3AS20%E2%80%9329%2E)
19. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013a. Annotating temporal information in clinical narratives. Journal of Biomedical Informatics, 46 Suppl:S5–12. [find](https://scholar.google.com/scholar?q=Weiyi+Sun%2C+Anna+Rumshisky%2C+and+Ozlem+Uzuner%2E+2013a%2E+Annotating+temporal+information+in+clinical+narratives%2E+Journal+of+Biomedical+Informatics%2C+46+Suppl%3AS5%E2%80%9312%2E)
20. Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner. 2013b. Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. Journal of the American Medical Informatics Association: JAMIA, 20(5):806–813. [find](https://scholar.google.com/scholar?q=Weiyi+Sun%2C+Anna+Rumshisky%2C+and+Ozlem+Uzuner%2E+2013b%2E+Evaluating+temporal+relations+in+clinical+text%3A+2012+i2b2+Challenge%2E+Journal+of+the+American+Medical+Informatics+Association%3A+JAMIA%2C+20%285%29%3A806%E2%80%93813%2E)
21. Ozlem Uzuner, Yuan Luo, and Peter Szolovits. 2007. Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association: JAMIA, 14(5):550–563. zlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/VA challenge on concepts, as… [find](https://scholar.google.com/scholar?q=Ozlem+Uzuner%2C+Yuan+Luo%2C+and+Peter+Szolovits%2E+2007%2E+Evaluating+the+state%2Dof%2Dthe%2Dart+in+automatic+de%2Didentification%2E+Journal+of+the+American+Medical+Informatics+Association%3A+JAMIA%2C+14%285%29%3A550%E2%80%93563%2E+zlem+U)
22. Henghui Zhu, Ioannis Ch Paschalidis, and Amir Tahmasebi. 2018. Clinical Concept Extraction with Contextual Word Embedding. arXiv:1810.10566 [cs]. ArXiv: 1810.10566. — [arXiv:1810.10566](https://arxiv.org/abs/1810.10566) [find](https://scholar.google.com/scholar?q=Henghui+Zhu%2C+Ioannis+Ch+Paschalidis%2C+and+Amir+Tahmasebi%2E+2018%2E+Clinical+Concept+Extraction+with+Contextual+Word+Embedding%2E+arXiv%3A1810%2E10566+%5Bcs%5D%2E+ArXiv%3A+1810%2E10566%2E+%E2%80%94+%5BarXiv%3A1810%2E10566%5D%28https%3A%2F%2Farxiv%2E)