---
title: "Biomedical and Clinical English Model Packages in the Stanza"
description: "General-purpose NLP toolkits handle biomedical and clinical English poorly… 68.99 LAS on CRAFT. 38 references, cited by 10 papers here."
authors: ["Yuhao Zhang", "Yuhui Zhang", "Peng Qi", "Christopher D. Manning", "Curtis P. Langlotz"]
year: 2020
venue: "arXiv:2007.14640"
source: "https://arxiv.org/abs/2007.14640"
doi: "10.48550/arxiv.2007.14640"
arxiv: "2007.14640"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2020-biomedical-clinical-english-model-packages/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2020-biomedical-clinical-english-model-packages/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/) / Biomedical and Clinical English Model Packages in the Stanza Python NLP Library

# Biomedical and Clinical English Model Packages in the Stanza Python NLP Library

**Authors:** [Yuhao Zhang](https://md.research.elamur.ai/free/en/authors/yuhao-zhang/), [Yuhui Zhang](https://md.research.elamur.ai/free/en/authors/yuhui-zhang/), [Peng Qi](https://md.research.elamur.ai/free/en/authors/peng-qi/), [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) and [Curtis P. Langlotz](https://md.research.elamur.ai/free/en/authors/curtis-p-langlotz/) · **Year:** 2020 · **Venue:** arXiv:2007.14640 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2007.14640) · [PDF](https://arxiv.org/pdf/2007.14640) · [DOI 10.48550/arxiv.2007.14640](https://doi.org/10.48550/arxiv.2007.14640) · [arXiv 2007.14640](https://arxiv.org/abs/2007.14640) · [OpenAlex](https://openalex.org/W3045493334)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · 10 in our library, of the 149,439 papers whose bibliographies we read

## Summary
General-purpose NLP toolkits handle biomedical and clinical English poorly, and domain-specific tools often lack modern neural models or native Python support. The authors extend the fully neural Stanza library with domain packages for both syntax and NER. For syntax, they convert the CRAFT and GENIA treebanks from Penn Treebank annotation into UDv2 with the CoreNLP converter and train pipelines covering tokenization, POS, lemmas and dependency parsing. No public clinical treebank exists, so they build a silver one. They tokenize MIMIC-III notes with CoreNLP and parse them with the general EWT-trained Stanza model. In-domain data is mixed with EWT to make the models more robust. For NER, they use a BiLSTM-CRF tagger with forward and backward character language models pretrained on PubMed or MIMIC-III text. The packages match or beat prior toolkits such as scispaCy and CoreNLP on syntax. NER comes close to BioBERT while running much faster. Everything is available through Stanza's ordinary Python interface.

## Key points
- Released resources:
  - two biomedical UD pipelines (CRAFT, GENIA);
  - one clinical UD pipeline (silver MIMIC);
  - 8 biomedical NER models (AnatEM, BC5CDR, BC4CHEMD, BioNLP13CG, JNLPBA, Linnaeus, NCBI-Disease, S800);
  - 2 clinical NER models (i2b2 and a new radiology-report model);
  - the UD-converted CRAFT and GENIA files.
- Table 1 LAS for Stanza is 89.67 on CRAFT versus 81.81 for CoreNLP and 77.74 for scispaCy. On GENIA it is 89.48 versus 83.16 and 86.57. On the silver MIMIC data it is 82.81.
- Treebank mixing for CRAFT:
  - A model trained only on EWT reaches just 68.99 LAS on CRAFT.
  - Combining EWT with CRAFT gives 89.57 LAS on CRAFT, while costing only about 1.2 LAS on the EWT test set.
- The silver MIMIC treebank uses 800 randomly sampled notes split 600/100/100, sized to resemble EWT. The biomedical CharLMs were pretrained on a PubMed dump of about 2.1 billion tokens.
- Pretrained domain CharLMs add on average +2.91 F1 on biomedical NER and +1.94 F1 on clinical NER, compared with a randomly initialized character BiLSTM. On GPU, Stanza NER runs at roughly scispaCy speed and far faster than BioBERT.

## Interesting ideas
- Bootstrap a treebank for a domain without gold data: run a strong rule-based tokenizer and a general-domain parser over raw domain text, then train on the resulting silver trees.
- Train the domain model on in-domain data concatenated with the general treebank, and always re-test on the general test set to measure what was lost.
- Treebanks may disagree on tokenization conventions, such as splitting hyphenated words (CRAFT) versus keeping them whole (GENIA). The authors only mixed EWT with the treebank whose tokenization matched it.
- Tokenization and sentence splitting are learned jointly as one tagging task over characters (end of token, end of sentence, or neither). The lemmatizer pairs a dictionary with a seq2seq model plus a shortcut classifier (lowercase or copy) for long odd strings like URLs.
- When pretraining the clinical CharLMs, sentences containing anonymization masks were dropped so placeholder patterns would not dominate the model. The value of pretraining was then measured against a randomly initialized ablation.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Unannotated genres such as fables, school tests and word problems could get silver UD trees this way. Ideally several witness parsers would agree first, and the silver set should stay roughly the size of EWT.
- **I2** (tree): When adapting the perceptron parser or gparse to a new genre, mixing in EWT and reporting the EWT score alongside guards against silent regressions.
- **I3** (tree): Before merging treebanks or converting UD dialects, check that their token conventions match. A mismatch could corrupt the training signal more than the extra data helps.
- **I4** (tokens): The crate en lemmatizer could use a similar dictionary-first design with an explicit copy/lowercase fallback. That keeps decisions explainable and makes junk tokens safe to handle.
- **I5** (learn): Filter templated or placeholder text out of pretraining corpora. Then prove each component's value against a no-pretraining control, in the spirit of negative-control evaluation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 234 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 231 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D. Manning and Curtis P. Langlotz. 2020. Biomedical and Clinical English Model Packages in the Stanza Python NLP Library. arXiv:2007.14640.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2020biomedical,
  title = {{Biomedical and Clinical English Model Packages in the Stanza Python NLP Library}},
  author = {Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D. Manning and Curtis P. Langlotz},
  year = {2020},
  doi = {10.48550/arxiv.2007.14640},
  eprint = {2007.14640},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.14640}
}
```

- [← previous Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models](https://md.research.elamur.ai/free/en/papers/schlegel-2020-leaderboards-survey-methods-revealing-weaknesses/)
- [next → Bridging Resolution: A Survey of the State of the Art](https://md.research.elamur.ai/free/en/papers/kobayashi-2020-bridging-resolution-survey-state-art/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/)

## References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. [Contextual string embeddings for sequence labeling](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/). In Proceedings of the 27th International Conference on Computational Linguistics.
2. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. [Globally normalized transition-based neural networks](https://md.research.elamur.ai/free/en/papers/andor-2016-globally-normalized-transition-based-neural/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
3. Alan R Aronson and François-Michel Lang. 2010. An overview of MetaMap: historical perspective and recent advances. Journal of the American Medical Informatics Association, 17(3):229–236. [↗](https://doi.org/10.1136/jamia.2009.002733)
4. William A Baumgartner Jr, Michael Bada, Sampo Pyysalo, Manuel R Ciosici, Negacy Hailu, Harrison Pielke-Lombardo, Michael Regan, and Lawrence Hunter. 2019. CRAFT shared tasks 2019 overview—integrated structure, semantics, and coreference. In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks. [find](https://scholar.google.com/scholar?q=William+A+Baumgartner+Jr%2C+Michael+Bada%2C+Sampo+Pyysalo%2C+Manuel+R+Ciosici%2C+Negacy+Hailu%2C+Harrison+Pielke%2DLombardo%2C+Michael+Regan%2C+and+Lawrence+Hunter%2E+2019%2E+CRAFT+shared+tasks+2019+overview%E2%80%94integrated+s)
5. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+2009%2E+Natural+language+processing+with+Python%3A+analyzing+text+with+the+natural+language+toolkit%2E+O%E2%80%99Reilly+Media%2C+Inc%2E)
6. YongGang Cao, Feifan Liu, Pippa Simpson, Lamont Antieau, Andrew Bennett, James J Cimino, John Ely, and Hong Yu. 2011. AskHERMES: An online question answering system for complex clinical questions. Journal of Biomedical Informatics, 44(2):277–288. [↗](https://doi.org/10.1016/j.jbi.2011.01.004)
7. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. [↗](https://doi.org/10.1016/j.jbi.2013.12.006)
8. Timothy Dozat and Christopher D. Manning. 2017. [Deep biaffine attention for neural dependency parsing](https://md.research.elamur.ai/free/en/papers/dozat-2016-deep-biaffine-attention-neural-dependency/). In International Conference on Learning Representations (ICLR).
9. Nan Du, Kai Chen, Anjuli Kannan, Linh Tran, Yuhui Chen, and Izhak Shafran. 2019. Extracting symptoms and their status from clinical conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nan+Du%2C+Kai+Chen%2C+Anjuli+Kannan%2C+Linh+Tran%2C+Yuhui+Chen%2C+and+Izhak+Shafran%2E+2019%2E+Extracting+symptoms+and+their+status+from+clinical+conversations%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Asso)
10. Martin Gerner, Goran Nenadic, and Casey M Bergman. 2010. LINNAEUS: a species name identification system for biomedical literature. BMC bioinformatics, 11(1):85. [↗](https://doi.org/10.1186/1471-2105-11-85)
11. Saeed Hassanpour and Curtis P Langlotz. 2016. Information extraction from multi-institutional radiology reports. Artificial intelligence in medicine, 66:29– 39. [↗](https://doi.org/10.1016/j.artmed.2015.09.007)
12. Lawrence Hunter and K Bretonnel Cohen. 2006. Biomedical language processing: what’s beyond PubMed? Molecular cell, 21(5):589–594. [↗](https://doi.org/10.1016/j.molcel.2006.02.012)
13. Ashish K Jha, Catherine M DesRoches, Eric G Campbell, Karen Donelan, Sowmya R Rao, Timothy G Ferris, Alexandra Shields, Sara Rosenbaum, and David Blumenthal. 2009. Use of electronic health records in US hospitals. New England Journal of Medicine, 360(16):1628–1638. [find](https://scholar.google.com/scholar?q=Ashish+K+Jha%2C+Catherine+M+DesRoches%2C+Eric+G+Campbell%2C+Karen+Donelan%2C+Sowmya+R+Rao%2C+Timothy+G+Ferris%2C+Alexandra+Shields%2C+Sara+Rosenbaum%2C+and+David+Blumenthal%2E+2009%2E+Use+of+electronic+health+records+in+)
14. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. [PubMedQA: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). [↗](https://doi.org/10.48550/arxiv.1909.06146)
15. Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. [MIMIC-III, a freely accessible critical care database](https://md.research.elamur.ai/free/en/papers/johnson-2016-mimic-iii-freely-accessible-critical/). Scientific data, 3(1):1–9.
16. J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003. GENIA corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, 19:i180– i182. [↗](https://doi.org/10.1093/bioinformatics/btg1023)
17. Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. 2004. [Introduction to the bio-entity recognition task at JNLPBA](https://md.research.elamur.ai/free/en/papers/collier-2004-introduction-bio-entity-recognition-task/). In Proceedings of the international joint workshop on natural language processing in biomedicine and its applications.
18. Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M Lowe, et al. 2015. The CHEMDNER corpus of chemicals and drugs and its annotation principles. Journal of Cheminformatics, 7(1):1–17. [↗](https://doi.org/10.1186/1758-2946-7-s1-s2)
19. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/). Bioinformatics, 36(4):1234–1240. [↗](https://doi.org/10.1093/bioinformatics/btz682)
20. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database: The Journal of Biological Databases and Curation, 2016. [↗](https://doi.org/10.1093/database/baw068)
21. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. [The Stanford CoreNLP natural language processing toolkit](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/). In Association for Computational Linguistics (ACL) System Demonstrations.
22. David McClosky and Eugene Charniak. 2008. Self-training for biomedical parsing. In Proceedings of ACL-08: HLT, Short Papers, pages 101–104. [find](https://scholar.google.com/scholar?q=David+McClosky+and+Eugene+Charniak%2E+2008%2E+Self%2Dtraining+for+biomedical+parsing%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+Short+Papers%2C+pages+101%E2%80%93104%2E)
23. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. [ScispaCy: Fast and robust models for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/neumann-2019-scispacy-fast-robust-models-biomedical/). Proceedings of the 18th BioNLP Workshop and Shared Task.
24. Thang Minh Ngo, Jenna Kanerva, Filip Ginter, and Sampo Pyysalo. 2019. Neural dependency parsing of biomedical text: TurkuNLP entry in the CRAFT structural annotation task. In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks. [find](https://scholar.google.com/scholar?q=Thang+Minh+Ngo%2C+Jenna+Kanerva%2C+Filip+Ginter%2C+and+Sampo+Pyysalo%2E+2019%2E+Neural+dependency+parsing+of+biomedical+text%3A+TurkuNLP+entry+in+the+CRAFT+structural+annotation+task%2E+In+Proceedings+of+The+5th+Wo)
25. Dat Quoc Nguyen and Karin Verspoor. 2019. From POS tagging to dependency parsing for biomedical event extraction. BMC Bioinformatics, 20(1):72. [↗](https://doi.org/10.1186/s12859-019-2604-0)
26. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. [Universal dependencies v2: An evergrowing multilingual treebank collection](https://md.research.elamur.ai/free/en/papers/nivre-2020-universal-dependencies-v2/). In Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC’20). [↗](https://doi.org/10.48550/arxiv.2004.10643)
27. Evangelos Pafilis, Sune P Frankild, Lucia Fanini, Sarah Faulwetter, Christina Pavloudi, Aikaterini Vasileiadou, Christos Arvanitidis, and Lars Juhl Jensen. 2013. The SPECIES and ORGANISMS resources for fast and accurate identification of taxonomic names in text. PLoS ONE, 8(6):e65390. [↗](https://doi.org/10.1371/journal.pone.0065390)
28. Hoifung Poon, Chris Quirk, Charlie DeZiel, and David Heckerman. 2014. Literome: PubMed-scale genomic knowledge base in the cloud. Bioinformatics, 30(19):2840–2842. [↗](https://doi.org/10.1093/bioinformatics/btu383)
29. Sampo Pyysalo and Sophia Ananiadou. 2014. Anatomical entity mention recognition at literature scale. Bioinformatics, 30(6):868–875. [↗](https://doi.org/10.1093/bioinformatics/btt580)
30. Sampo Pyysalo, Tomoko Ohta, Rafal Rak, Andrew Rowley, Hong-Woo Chun, Sung-Jae Jung, Sung-Pil Choi, Jun’ichi Tsujii, and Sophia Ananiadou. 2015. Overview of the cancer genetics and pathway curation tasks of BioNLP shared task 2013. BMC Bioinformatics, 16(S10):S2. [↗](https://doi.org/10.1186/1471-2105-16-s10-s2)
31. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the [CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies](https://md.research.elamur.ai/free/en/papers/zeman-2018-conll-2018-shared-task-multilingual/).
32. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. [Stanza: A Python natural language processing toolkit for many human languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. [↗](https://doi.org/10.18653/v1/2020.acl-demos.14)
33. Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. 2010. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informatics Association, 17(5):507–513. [↗](https://doi.org/10.1136/jamia.2009.001560)
34. Sebastian Schuster and Christopher D Manning. 2016. [Enhanced English universal dependencies: An improved representation for natural language understanding tasks](https://md.research.elamur.ai/free/en/papers/schuster-2016-enhanced-english-universal-dependencies-improved/). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2371–2378. [↗](https://doi.org/10.63317/3fjxjut9audq)
35. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). [↗](https://doi.org/10.63317/4esbhdz7y5u5)
36. Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556. [↗](https://doi.org/10.1136/amiajnl-2011-000203)
37. Karin Verspoor, Kevin Bretonnel Cohen, Arrick Lanfranchi, Colin Warner, Helen L Johnson, Christophe Roeder, Jinho D Choi, Christopher Funk, Yuriy Malenkiy, Miriam Eckert, et al. 2012. A corpus of full-text journal articles is a robust evaluation tool for revealing differences in performance of biomedical natural language processing tools. BMC bioinformatics, 13(1):207. [↗](https://doi.org/10.1186/1471-2105-13-207)
38. Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. 2019. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, 35(10):1745–1752. [↗](https://doi.org/10.1093/bioinformatics/bty869)

*Updated 1 November 2026*