---
title: "Combining Automatic Labelers and Expert — Smit 2020"
description: "Large chest x-ray datasets get their training labels from free-text radiology reports."
authors: ["Akshay Smit", "Saahil Jain", "Pranav Rajpurkar", "Anuj Pareek", "Andrew Ng", "Matthew Lungren"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.117/"
doi: "10.18653/v1/2020.emnlp-main.117"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/smit-2020-combining-automatic-labelers-expert-annotations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/smit-2020-combining-automatic-labelers-expert-annotations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT

# Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT

**Authors:** Akshay Smit, Saahil Jain, [Pranav Rajpurkar](https://md.research.elamur.ai/free/en/authors/pranav-rajpurkar/), [Anuj Pareek](https://md.research.elamur.ai/free/en/authors/anuj-pareek/), [Andrew Ng](https://md.research.elamur.ai/free/en/authors/andrew-ng/), Matthew Lungren · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.117/) · [PDF](https://aclanthology.org/2020.emnlp-main.117.pdf) · [DOI 10.18653/v1/2020.emnlp-main.117](https://doi.org/10.18653/v1/2020.emnlp-main.117) · [OpenAlex](https://openalex.org/W3103694015)

**Citations:** 250 citations (openalex-snapshot · 2026-10-14) · 84 in our library, of the 140884 papers whose bibliographies we read

## Summary
Large chest x-ray datasets get their training labels from free-text radiology reports. Older labelers rely on hand-written rules, which miss much of the nuance in clinical language. Newer Transformer labelers instead need many costly expert annotations. The authors combine the two sources. They start from a biomedically pretrained BERT with 14 per-observation output heads and train it on about 190k CheXpert reports labeled automatically by the rule-based CheXpert labeler. They then fine-tune it on 1,000 radiologist-annotated reports, enlarged with backtranslated paraphrases. The resulting model, CheXbert, labels each of 14 observations as blank, positive, negative or uncertain. It was tested on reports from an institution not seen in training, the MIMIC-CXR test set. There it beats the CheXpert labeler with statistical significance and comes within a small margin of a radiologist benchmark. This matters for any field with expensive expert labels: an existing rule system can provide large-scale pretraining signal, and a small expert set can then correct its errors.

## Key points
- **Data:**
  - Training used 190,460 deduplicated CheXpert reports labeled by the CheXpert labeler, plus the 1,000-report CheXpert manual set annotated by two board-certified radiologists.
  - Evaluation used 687 expert-labeled MIMIC-CXR test reports.
- **Architecture:**
  - BERT-base (~110M parameters) with 14 linear heads (~40k parameters) attached to the final CLS representation.
  - 12 heads cover abnormalities, 1 covers support devices and 1 covers No Finding.
  - Input is capped at 512 tokens.
- **Training:**
  - All layers are fine-tuned with Adam at a learning rate of 2×10⁻⁵ and a batch size of 18.
  - The losses of all 14 observations are summed.
  - The saved checkpoint is the one with the best average dev score.
- **Main result:** CheXbert beats the CheXpert labeler on MIMIC-CXR by 0.055 F1 (95% CI 0.039–0.070) and is 0.007 F1 below the radiologist benchmark.
- **Comparisons:** The paper contrasts three supervision regimes:
  - T-rad, trained only on expert labels;
  - T-auto, trained only on rule-based labels;
  - hybrid models trained on both.
  
  Frozen-encoder baselines are also included, using either CLS pooling or averaged tokens.

## Interesting ideas
- **Distill a rule system, then correct it with experts.** Training a neural model on a rule-based labeler's output first, and only then fine-tuning on a few hundred to a thousand gold examples, could suit any task where rules already exist, such as low-resource NER or classification.
- **Backtranslation as cheap augmentation for a tiny gold set.** Paraphrasing expert-labeled texts through another language and back adds lexical variety, for example swapping technical terms for everyday synonyms.
  - The authors also show its risks: it can produce non-words or drop important attributes.
  - Having a domain expert check a sample of the paraphrases is a worthwhile safeguard.
- **One shared encoder with many small classification heads.** Each target gets its own head, and the per-target losses are summed. This handles multi-label, multi-class extraction with almost no extra parameters.
- **Evaluation design.**
  - Each label type (positive, negative, uncertain) is scored as its own retrieval task, and the F1 scores are weighted by support.
  - Bootstrap confidence intervals are reported.
  - The test set comes from a different institution.
  
  Together these give a fairer picture than plain accuracy on in-domain data.
- **Qualitative error tables.** Placing the rule-based system, the distilled model and the final model side by side on specific phrases makes clear what each stage adds. Examples include misspellings, hedged wording and negation conveyed by words like "removal".

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{smit2020combining,
  title = {{Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT}},
  author = {Smit, Akshay and Jain, Saahil and Rajpurkar, Pranav and Pareek, Anuj and Ng, Andrew and Lungren, Matthew},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.117},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.117}
}
```

- [← previous ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/)
- [next → Combining Event Semantics and Degree Semantics for Natural Language Inference](https://md.research.elamur.ai/free/en/papers/haruta-2020-combining-event-semantics-degree-semantics/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

39 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly Available Clinical BERT Embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association for Computati… [find](https://scholar.google.com/scholar?q=Emily+Alsentzer%2C+John+Murphy%2C+William+Boag%2C+Wei%2DHung+Weng%2C+Di+Jindi%2C+Tristan+Naumann%2C+and+Matthew+McDermott%2E+2019%2E+Publicly+Available+Clinical+BERT+Embeddings%2E+In+Proceedings+of+the+2nd+Clinical+Natur) [our card](https://md.research.elamur.ai/free/en/papers/alsentzer-2019-publicly-available-clinical-bert-embeddings/)
2. Mauro Annarumma, Samuel J. Withey, Robert J. Bakewell, Emanuele Pesce, Vicky Goh, and Giovanni Montana. 2019. Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks. Radiology, 291(1):196–202. Publisher: Radiological Society of North America. [find](https://scholar.google.com/scholar?q=Mauro+Annarumma%2C+Samuel+J%2E+Withey%2C+Robert+J%2E+Bakewell%2C+Emanuele+Pesce%2C+Vicky+Goh%2C+and+Giovanni+Montana%2E+2019%2E+Automated+Triaging+of+Adult+Chest+Radiographs+with+Deep+Artificial+Neural+Networks%2E+Radiol)
3. Imon Banerjee, Matthew C. Chen, Matthew P. Lungren, and Daniel L. Rubin. 2018. Radiology report annotation using intelligent word embeddings: Applied to multi-institutional chest CT cohort. Journal of Biomedical Informatics, 77:11–20. [find](https://scholar.google.com/scholar?q=Imon+Banerjee%2C+Matthew+C%2E+Chen%2C+Matthew+P%2E+Lungren%2C+and+Daniel+L%2E+Rubin%2E+2018%2E+Radiology+report+annotation+using+intelligent+word+embeddings%3A+Applied+to+multi%2Dinstitutional+chest+CT+cohort%2E+Journal+of)
4. Olivier Bodenreider. 2004. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Research, 32(Database issue):D267–D270. [find](https://scholar.google.com/scholar?q=Olivier+Bodenreider%2E+2004%2E+The+Unified+Medical+Language+System+%28UMLS%29%3A+integrating+biomedical+terminology%2E+Nucleic+Acids+Research%2C+32%28Database+issue%29%3AD267%E2%80%93D270%2E)
5. Selen Bozkurt, Emel Alkim, Imon Banerjee, and Daniel L. Rubin. 2019. Automated Detection of Measurements and Their Descriptors in Radiology Reports Using a Hybrid Natural Language Processing Algorithm. Journal of Digital Imaging, 32(4):544–553. [find](https://scholar.google.com/scholar?q=Selen+Bozkurt%2C+Emel+Alkim%2C+Imon+Banerjee%2C+and+Daniel+L%2E+Rubin%2E+2019%2E+Automated+Detection+of+Measurements+and+Their+Descriptors+in+Radiology+Reports+Using+a+Hybrid+Natural+Language+Processing+Algorithm)
6. Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá. 2019. PadChest: A large chest x-ray image dataset with multi-label annotated reports. arXiv:1901.07441 [cs, eess]. ArXiv: 1901.07441. — [arXiv:1901.07441](https://arxiv.org/abs/1901.07441) [find](https://scholar.google.com/scholar?q=Aurelia+Bustos%2C+Antonio+Pertusa%2C+Jose%2DMaria+Salinas%2C+and+Maria+de+la+Iglesia%2DVaya%CC%81%2E+2019%2E+PadChest%3A+A+large+chest+x%2Dray+image+dataset+with+multi%2Dlabel+annotated+reports%2E+arXiv%3A1901%2E07441+%5Bcs%2C+eess%5D%2E+A)
7. Alison Callahan, Jason A. Fries, Christopher Ré, James I. Huddleston, Nicholas J. Giori, Scott Delp, and Nigam H. Shah. 2019. Medical device surveillance with electronic health records. npj Digital Medicine, 2(1):1–10. Number: 1 Publisher: Nature Publishing Group. [find](https://scholar.google.com/scholar?q=Alison+Callahan%2C+Jason+A%2E+Fries%2C+Christopher+Re%CC%81%2C+James+I%2E+Huddleston%2C+Nicholas+J%2E+Giori%2C+Scott+Delp%2C+and+Nigam+H%2E+Shah%2E+2019%2E+Medical+device+surveillance+with+electronic+health+records%2E+npj+Digital+M)
8. Brian E. Chapman, Sean Lee, Hyunseok Peter Kang, and Wendy W. Chapman. 2011. Document-level classification of CT pulmonary angiography reports based on an extension of the ConText algorithm. Journal of Biomedical Informatics, 44(5):728–737. [find](https://scholar.google.com/scholar?q=Brian+E%2E+Chapman%2C+Sean+Lee%2C+Hyunseok+Peter+Kang%2C+and+Wendy+W%2E+Chapman%2E+2011%2E+Document%2Dlevel+classification+of+CT+pulmonary+angiography+reports+based+on+an+extension+of+the+ConText+algorithm%2E+Journal+o)
9. Wendy W. Chapman, Will Bridewell, Paul Hanbury, Gregory F. Cooper, and Bruce G. Buchanan. 2001. A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries. Journal of Biomedical Informatics, 34(5):301–310. [find](https://scholar.google.com/scholar?q=Wendy+W%2E+Chapman%2C+Will+Bridewell%2C+Paul+Hanbury%2C+Gregory+F%2E+Cooper%2C+and+Bruce+G%2E+Buchanan%2E+2001%2E+A+Simple+Algorithm+for+Identifying+Negated+Findings+and+Diseases+in+Discharge+Summaries%2E+Journal+of+Biom)
10. Matthew C. Chen, Robyn L. Ball, Lingyao Yang, Nathaniel Moradzadeh, Brian E. Chapman, David B. Larson, Curtis P. Langlotz, Timothy J. Amrhein, and Matthew P. Lungren. 2017. Deep Learning to Classify Radiology Free-Text Reports. Radiology, 286(3):845–852. Publisher: Radiological Society of North Amer… [find](https://scholar.google.com/scholar?q=Matthew+C%2E+Chen%2C+Robyn+L%2E+Ball%2C+Lingyao+Yang%2C+Nathaniel+Moradzadeh%2C+Brian+E%2E+Chapman%2C+David+B%2E+Larson%2C+Curtis+P%2E+Langlotz%2C+Timothy+J%2E+Amrhein%2C+and+Matthew+P%2E+Lungren%2E+2017%2E+Deep+Learning+to+Classify+R)
11. Po-Hao Chen, Hanna Zafar, Maya Galperin-Aizenberg, and Tessa Cook. 2018. Integrating Natural Language Processing and Machine Learning Algorithms to Categorize Oncologic Response in Radiology Reports. Journal of Digital Imaging, 31(2):178–184. [find](https://scholar.google.com/scholar?q=Po%2DHao+Chen%2C+Hanna+Zafar%2C+Maya+Galperin%2DAizenberg%2C+and+Tessa+Cook%2E+2018%2E+Integrating+Natural+Language+Processing+and+Machine+Learning+Algorithms+to+Categorize+Oncologic+Response+in+Radiology+Reports%2E+)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. ArXiv: 1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+arXiv%3A1810%2E04805+%5Bcs%5D%2E+ArXiv%3A+1810%2E04805%2E)
13. Ignat Drozdov, Daniel Forbes, Benjamin Szubert, Mark Hall, Chris Carlin, and David J. Lowe. 2020. Supervised and unsupervised language modelling in Chest X-Ray radiological reports. PLOS ONE, 15(3):e0229963. Publisher: Public Library of Science. [find](https://scholar.google.com/scholar?q=Ignat+Drozdov%2C+Daniel+Forbes%2C+Benjamin+Szubert%2C+Mark+Hall%2C+Chris+Carlin%2C+and+David+J%2E+Lowe%2E+2020%2E+Supervised+and+unsupervised+language+modelling+in+Chest+X%2DRay+radiological+reports%2E+PLOS+ONE%2C+15%283%29%3Ae0)
14. Jared Dunnmon, Alexander Ratner, Nishith Khandwala, Khaled Saab, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew Lungren, Daniel Rubin, and Christopher Ré. 2019. Cross-Modal Data Programming Enables Rapid Medical Machine Learning. arXiv:1903.11101 [cs, eess, stat]. A… — [arXiv:1903.11101](https://arxiv.org/abs/1903.11101) [find](https://scholar.google.com/scholar?q=Jared+Dunnmon%2C+Alexander+Ratner%2C+Nishith+Khandwala%2C+Khaled+Saab%2C+Matthew+Markert%2C+Hersh+Sagreiya%2C+Roger+Goldman%2C+Christopher+Lee%2DMesser%2C+Matthew+Lungren%2C+Daniel+Rubin%2C+and+Christopher+Re%CC%81%2E+2019%2E+Cross)
15. B. Efron and R. Tibshirani. 1986. Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy. Statistical Science, 1(1):54–75. Publisher: Institute of Mathematical Statistics. [find](https://scholar.google.com/scholar?q=B%2E+Efron+and+R%2E+Tibshirani%2E+1986%2E+Bootstrap+Methods+for+Standard+Errors%2C+Confidence+Intervals%2C+and+Other+Measures+of+Statistical+Accuracy%2E+Statistical+Science%2C+1%281%29%3A54%E2%80%9375%2E+Publisher%3A+Institute+of+Math)
16. Saeed Hassanpour, Curtis P. Langlotz, Timothy J. Amrhein, Nicholas T. Befera, and Matthew P. Lungren. 2017. Performance of a Machine Learning Classifier of Knee MRI Reports in Two Large Academic Radiology Practices: A Tool to Estimate Diagnostic Yield. American Journal of Roentgenology, 208(4):750–7… [find](https://scholar.google.com/scholar?q=Saeed+Hassanpour%2C+Curtis+P%2E+Langlotz%2C+Timothy+J%2E+Amrhein%2C+Nicholas+T%2E+Befera%2C+and+Matthew+P%2E+Lungren%2E+2017%2E+Performance+of+a+Machine+Learning+Classifier+of+Knee+MRI+Reports+in+Two+Large+Academic+Radio)
17. Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lu… — [arXiv:1901.07031](https://arxiv.org/abs/1901.07031) [find](https://scholar.google.com/scholar?q=Jeremy+Irvin%2C+Pranav+Rajpurkar%2C+Michael+Ko%2C+Yifan+Yu%2C+Silviana+Ciurea%2DIlcus%2C+Chris+Chute%2C+Henrik+Marklund%2C+Behzad+Haghgoo%2C+Robyn+Ball%2C+Katie+Shpanskaya%2C+Jayne+Seekins%2C+David+A%2E+Mong%2C+Safwan+S%2E+Halabi%2C)
18. Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth J. Berkowitz, and Steven Horng. 2019. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv:1901.07042 [cs, eess]. ArXiv: 1… — [arXiv:1901.07042](https://arxiv.org/abs/1901.07042) [find](https://scholar.google.com/scholar?q=Alistair+E%2E+W%2E+Johnson%2C+Tom+J%2E+Pollard%2C+Nathaniel+R%2E+Greenbaum%2C+Matthew+P%2E+Lungren%2C+Chih%2Dying+Deng%2C+Yifan+Peng%2C+Zhiyong+Lu%2C+Roger+G%2E+Mark%2C+Seth+J%2E+Berkowitz%2C+and+Steven+Horng%2E+2019%2E+MIMIC%2DCXR%2DJPG%2C+a+l)
19. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, page btz682. ArXiv: 1901.08746. [find](https://scholar.google.com/scholar?q=Jinhyuk+Lee%2C+Wonjin+Yoon%2C+Sungdong+Kim%2C+Donghyeon+Kim%2C+Sunkyu+Kim%2C+Chan+Ho+So%2C+and+Jaewoo+Kang%2E+2019%2E+BioBERT%3A+a+pre%2Dtrained+biomedical+language+representation+model+for+biomedical+text+mining%2E+Bioinf) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
20. Matthew B. A. McDermott, Tzu Ming Harry Hsu, Wei-Hung Weng, Marzyeh Ghassemi, and Peter Szolovits. 2020. Chexpert++: Approximating the chexpert labeler for speed,differentiability, and probabilistic output. [find](https://scholar.google.com/scholar?q=Matthew+B%2E+A%2E+McDermott%2C+Tzu+Ming+Harry+Hsu%2C+Wei%2DHung+Weng%2C+Marzyeh+Ghassemi%2C+and+Peter+Szolovits%2E+2020%2E+Chexpert%2B%2B%3A+Approximating+the+chexpert+labeler+for+speed%2Cdifferentiability%2C+and+probabilistic+o)
21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pa… [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+Representations+of+Words+and+Phrases+and+their+Compositionality%2E+In+C%2E+J%2E+C%2E+Burges%2C+L%2E+Bottou%2C+M%2E+Welling%2C+Z%2E)
22. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR’s WMT19 News Translation Task Submission. arXiv:1907.06616 [cs]. ArXiv: 1907.06616. [find](https://scholar.google.com/scholar?q=Nathan+Ng%2C+Kyra+Yee%2C+Alexei+Baevski%2C+Myle+Ott%2C+Michael+Auli%2C+and+Sergey+Edunov%2E+2019%2E+Facebook+FAIR%E2%80%99s+WMT19+News+Translation+Task+Submission%2E+arXiv%3A1907%2E06616+%5Bcs%5D%2E+ArXiv%3A+1907%2E06616%2E) [our card](https://md.research.elamur.ai/free/en/papers/ng-2019-facebook-fair-s-wmt19-news/)
23. Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong Lu. 2017. NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. arXiv:1712.05898 [cs]. ArXiv: 1712.05898. [find](https://scholar.google.com/scholar?q=Yifan+Peng%2C+Xiaosong+Wang%2C+Le+Lu%2C+Mohammadhadi+Bagheri%2C+Ronald+Summers%2C+and+Zhiyong+Lu%2E+2017%2E+NegBio%3A+a+high%2Dperformance+tool+for+negation+and+uncertainty+detection+in+radiology+reports%2E+arXiv%3A1712%2E05)
24. Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. arXiv:1906.05474 [cs]. ArXiv: 1906.05474. [find](https://scholar.google.com/scholar?q=Yifan+Peng%2C+Shankai+Yan%2C+and+Zhiyong+Lu%2E+2019%2E+Transfer+Learning+in+Biomedical+Natural+Language+Processing%3A+An+Evaluation+of+BERT+and+ELMo+on+Ten+Benchmarking+Datasets%2E+arXiv%3A1906%2E05474+%5Bcs%5D%2E+ArXiv%3A+1) [our card](https://md.research.elamur.ai/free/en/papers/peng-2019-transfer-learning-biomedical-natural-language/)
25. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+Vectors+for+Word+Representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Process)
26. Alberto Poncelas, Dimitar Shterionov, Andy Way, Gideon Maillette de Buy Wenniger, and Peyman Passban. 2018. Investigating Backtranslation in Neural Machine Translation. arXiv:1804.06189 [cs]. ArXiv: 1804.06189. [find](https://scholar.google.com/scholar?q=Alberto+Poncelas%2C+Dimitar+Shterionov%2C+Andy+Way%2C+Gideon+Maillette+de+Buy+Wenniger%2C+and+Peyman+Passban%2E+2018%2E+Investigating+Backtranslation+in+Neural+Machine+Translation%2E+arXiv%3A1804%2E06189+%5Bcs%5D%2E+ArXiv%3A+1) [our card](https://md.research.elamur.ai/free/en/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/)
27. Ewoud Pons, Loes M. M. Braun, M. G. Myriam Hunink, and Jan A. Kors. 2016. Natural Language Processing in Radiology: A Systematic Review. Radiology, 279(2):329–343. Publisher: Radiological Society of North America. [find](https://scholar.google.com/scholar?q=Ewoud+Pons%2C+Loes+M%2E+M%2E+Braun%2C+M%2E+G%2E+Myriam+Hunink%2C+and+Jan+A%2E+Kors%2E+2016%2E+Natural+Language+Processing+in+Radiology%3A+A+Systematic+Review%2E+Radiology%2C+279%282%29%3A329%E2%80%93343%2E+Publisher%3A+Radiological+Society+of+N)
28. Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré. 2018. Snorkel MeTaL: Weak Supervision for Multi-Task Learning. In Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning, DEEM’18, pages 1–4, Houston, TX, USA. Association for Computing Mac… [find](https://scholar.google.com/scholar?q=Alex+Ratner%2C+Braden+Hancock%2C+Jared+Dunnmon%2C+Roger+Goldman%2C+and+Christopher+Re%CC%81%2E+2018%2E+Snorkel+MeTaL%3A+Weak+Supervision+for+Multi%2DTask+Learning%2E+In+Proceedings+of+the+Second+Workshop+on+Data+Management+)
29. Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2020. Snorkel: rapid training data creation with weak supervision. The VLDB Journal, 29(2):709–730. [find](https://scholar.google.com/scholar?q=Alexander+Ratner%2C+Stephen+H%2E+Bach%2C+Henry+Ehrenberg%2C+Jason+Fries%2C+Sen+Wu%2C+and+Christopher+Re%CC%81%2E+2020%2E+Snorkel%3A+rapid+training+data+creation+with+weak+supervision%2E+The+VLDB+Journal%2C+29%282%29%3A709%E2%80%93730%2E)
30. Khaled Saab, Jared Dunnmon, Roger Goldman, Alex Ratner, Hersh Sagreiya, Christopher Ré, and Daniel Rubin. 2019. Doubly Weak Supervision of Deep Learning Models for Head CT. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Lecture Notes in Computer Science, pages 811–819,… [find](https://scholar.google.com/scholar?q=Khaled+Saab%2C+Jared+Dunnmon%2C+Roger+Goldman%2C+Alex+Ratner%2C+Hersh+Sagreiya%2C+Christopher+Re%CC%81%2C+and+Daniel+Rubin%2E+2019%2E+Doubly+Weak+Supervision+of+Deep+Learning+Models+for+Head+CT%2E+In+Medical+Image+Computing)
31. Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. 2010. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informati… [find](https://scholar.google.com/scholar?q=Guergana+K+Savova%2C+James+J+Masanz%2C+Philip+V+Ogren%2C+Jiaping+Zheng%2C+Sunghwan+Sohn%2C+Karin+C+Kipper%2DSchuler%2C+and+Christopher+G+Chute%2E+2010%2E+Mayo+clinical+Text+Analysis+and+Knowledge+Extraction+System+%28cTA)
32. Cole, and Thomas C. Booth. 2020. Automated Labelling using an Attention model for Radiology reports of MRI scans (ALARM). arXiv:2002.06588 [cs]. ArXiv: 2002.06588. — [arXiv:2002.06588](https://arxiv.org/abs/2002.06588) [find](https://scholar.google.com/scholar?q=Cole%2C+and+Thomas+C%2E+Booth%2E+2020%2E+Automated+Labelling+using+an+Attention+model+for+Radiology+reports+of+MRI+scans+%28ALARM%29%2E+arXiv%3A2002%2E06588+%5Bcs%5D%2E+ArXiv%3A+2002%2E06588%2E+%E2%80%94+%5BarXiv%3A2002%2E06588%5D%28https%3A%2F%2Farxiv%2Eo)
33. Kui Xue, Yangming Zhou, Zhiyuan Ma, Tong Ruan, Huanhuan Zhang, and Ping He. 2019. Fine-tuning BERT for Joint Entity and Relation Extraction in Chinese Medical Text. [find](https://scholar.google.com/scholar?q=Kui+Xue%2C+Yangming+Zhou%2C+Zhiyuan+Ma%2C+Tong+Ruan%2C+Huanhuan+Zhang%2C+and+Ping+He%2E+2019%2E+Fine%2Dtuning+BERT+for+Joint+Entity+and+Relation+Extraction+in+Chinese+Medical+Text%2E)
34. Kabir Yadav, Efsun Sarioglu, Hyeong Ah Choi, Walter B. Cartwright, Pamela S. Hinds, and James M. Chamberlain. 2016. Automated Outcome Classification of Computed Tomography Imaging Reports for Pediatric Traumatic Brain Injury. Academic Emergency Medicine: Official Journal of the Society for Academic… [find](https://scholar.google.com/scholar?q=Kabir+Yadav%2C+Efsun+Sarioglu%2C+Hyeong+Ah+Choi%2C+Walter+B%2E+Cartwright%2C+Pamela+S%2E+Hinds%2C+and+James+M%2E+Chamberlain%2E+2016%2E+Automated+Outcome+Classification+of+Computed+Tomography+Imaging+Reports+for+Pediatri)
35. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv:1906.08237 [cs]. ArXiv: 1906.08237. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V%2E+Le%2E+2020%2E+XLNet%3A+Generalized+Autoregressive+Pretraining+for+Language+Understanding%2E+arXiv%3A1906%2E08237+%5Bcs%5D%2E+ArXi) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
36. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. arXiv:1804.09541 [cs]. ArXiv: 1804.09541. [find](https://scholar.google.com/scholar?q=Adams+Wei+Yu%2C+David+Dohan%2C+Minh%2DThang+Luong%2C+Rui+Zhao%2C+Kai+Chen%2C+Mohammad+Norouzi%2C+and+Quoc+V%2E+Le%2E+2018%2E+QANet%3A+Combining+Local+Convolution+with+Global+Self%2DAttention+for+Reading+Comprehension%2E+arXiv%3A) [our card](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)
37. John Zech, Margaret Pain, Joseph Titano, Marcus Badgeley, Javin Schefflein, Andres Su, Anthony Costa, Joshua Bederson, Joseph Lehar, and Eric Karl Oermann. 2018. Natural Language–based Machine Learning Models for the Annotation of Clinical Radiology Reports. Radiology, 287(2):570–580. Publisher: Rad… [find](https://scholar.google.com/scholar?q=John+Zech%2C+Margaret+Pain%2C+Joseph+Titano%2C+Marcus+Badgeley%2C+Javin+Schefflein%2C+Andres+Su%2C+Anthony+Costa%2C+Joshua+Bederson%2C+Joseph+Lehar%2C+and+Eric+Karl+Oermann%2E+2018%2E+Natural+Language%E2%80%93based+Machine+Learnin)
38. Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. 2017. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. 2017 IEEE Conference on Computer Vision and Pattern Recogni… — [arXiv:1705.02315](https://arxiv.org/abs/1705.02315) [find](https://scholar.google.com/scholar?q=Xiaosong+Wang%2C+Yifan+Peng%2C+Le+Lu%2C+Zhiyong+Lu%2C+Mohammadhadi+Bagheri%2C+and+Ronald+M%2E+Summers%2E+2017%2E+ChestX%2Dray8%3A+Hospital%2Dscale+Chest+X%2Dray+Database+and+Benchmarks+on+Weakly%2DSupervised+Classification+and)
39. Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, and Hongfang Liu. 2018. Clinical information extraction applications: A literature review. Journal of Biomedical Informatics, 77:34–49. [find](https://scholar.google.com/scholar?q=Yanshan+Wang%2C+Liwei+Wang%2C+Majid+Rastegar%2DMojarad%2C+Sungrim+Moon%2C+Feichen+Shen%2C+Naveed+Afzal%2C+Sijia+Liu%2C+Yuqun+Zeng%2C+Saeed+Mehrabi%2C+Sunghwan+Sohn%2C+and+Hongfang+Liu%2E+2018%2E+Clinical+information+extraction)