---
title: "Self-Alignment Pretraining for Biomedical Entity — Liu 2021"
description: "The paper addresses a known weakness of biomedical masked language models such as PubMedBERT."
authors: ["Fangyu Liu", "Ehsan Shareghi", "Zaiqiao Meng", "Marco Basaldella", "Nigel Collier"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.334/"
doi: "10.18653/v1/2021.naacl-main.334"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2021-self-alignment-pretraining-biomedical-entity/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2021-self-alignment-pretraining-biomedical-entity/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Self-Alignment Pretraining for Biomedical Entity Representations

# Self-Alignment Pretraining for Biomedical Entity Representations

**Authors:** [Fangyu Liu](https://md.research.elamur.ai/free/en/authors/fangyu-liu/), [Ehsan Shareghi](https://md.research.elamur.ai/free/en/authors/ehsan-shareghi/), [Zaiqiao Meng](https://md.research.elamur.ai/free/en/authors/zaiqiao-meng/), Marco Basaldella, [Nigel Collier](https://md.research.elamur.ai/free/en/authors/nigel-collier/) · **Year:** 2021 · **Venue:** Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.334/) · [PDF](https://aclanthology.org/2021.naacl-main.334.pdf) · [DOI 10.18653/v1/2021.naacl-main.334](https://doi.org/10.18653/v1/2021.naacl-main.334) · [OpenAlex](https://openalex.org/W3164540570)

**Citations:** 335 citations (openalex-snapshot · 2026-10-14) · [97](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

The paper addresses a known weakness of biomedical masked language models such as PubMedBERT. They do not place different surface names of the same concept close together, which hurts medical entity linking (MEL). For example, Hydroxychloroquine, HCQ and Plaquenil should map to one concept. The authors propose SapBERT, which continues pretraining a BERT model on synonym sets from UMLS 2020AA so that names sharing a CUI form tight clusters. Each name is encoded through its [CLS] vector. Training uses online hard-pair mining within each mini-batch and an adapted Multi-Similarity loss. At inference, linking requires only nearest-neighbour search over these embeddings, with no hand-crafted pre- or post-processing.

Across six MEL benchmarks, the approach improves every BERT variant tested. Built on PubMedBERT, it sets new state-of-the-art results. On scientific-text datasets, it beats earlier supervised hybrid systems without any task-specific training. On social media data, it needs fine-tuning to lead. The work shows that a single, simple model can replace complex multi-stage linking pipelines.

## Key points

- **Pretraining data:** UMLS 2020AA has over 4M concepts and over 10M synonyms from more than 150 vocabularies, including MeSH, SNOMED CT and RxNorm. The authors extracted English names, lowercased them and added drug tradenames, giving 9,712,959 (name, CUI) entries. These were turned into 11,792,953 synonym pairs, with at most 50 pairs per concept.
- **Training cost:** One epoch (about 50k iterations, batch size 512) took roughly 5 hours using AdamW and mixed precision.
- **Benchmarks and results:** The six datasets are NCBI, BC5CDR-d, BC5CDR-c, MedMentions, AskAPatient and COMETA. SapBERT is significantly better than prior SOTA on five of them. On BC5CDR-c it essentially ties (96.5 vs. 96.6 Acc@1). Gains over base *BERT models reach 31.0 points of Acc@1 on social media data.
- **Ablation:** Turning off hard-pair mining on COMETA (zero-shot) drops Acc@1 from 67.2 to 52.3.
- **ADAPTER variants:** Versions that train only 13.22% or 1.09% of the parameters match full training on scientific datasets but fall behind on social media. Initialising BioSyn with SapBERT weights also improves BioSyn's results.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

97 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 95 not yet.

- [Coreference Resolution for the Biomedical Domain: A Survey](https://md.research.elamur.ai/free/en/papers/lu-2021-coreference-resolution-biomedical-domain-survey/)
- [QA-KGNet: 一种语言模型驱动的知识图谱问答模型](https://md.research.elamur.ai/free/en/papers/qiao-2023-qa-kgnet-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2021self,
  title = {{Self-Alignment Pretraining for Biomedical Entity Representations}},
  author = {Liu, Fangyu and Shareghi, Ehsan and Meng, Zaiqiao and Basaldella, Marco and Collier, Nigel},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.334},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.334}
}
```

- [← previous Scaling Within Document Coreference to Long Texts](https://md.research.elamur.ai/free/en/papers/thirukovalluru-2021-scaling-within-document-coreference-long/)
- [next → Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

50 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kevin Donnelly. 2006. SNOMED-CT: The advanced terminology and coding system for eHealth. Studies in health technology and informatics, 121:279. [find](https://scholar.google.com/scholar?q=Kevin+Donnelly%2E+2006%2E+SNOMED%2DCT%3A+The+advanced+terminology+and+coding+system+for+eHealth%2E+Studies+in+health+technology+and+informatics%2C+121%3A279%2E)
2. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association for Computati… [find](https://scholar.google.com/scholar?q=Emily+Alsentzer%2C+John+Murphy%2C+William+Boag%2C+Wei%2DHung+Weng%2C+Di+Jindi%2C+Tristan+Naumann%2C+and+Matthew+McDermott%2E+2019%2E+Publicly+available+clinical+BERT+embeddings%2E+In+Proceedings+of+the+2nd+Clinical+Natur) [our card](https://md.research.elamur.ai/free/en/papers/alsentzer-2019-publicly-available-clinical-bert-embeddings/)
3. Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013. How noisy social media text, how diffrnt social media sources? In Proceedings of the Sixth International Joint Conference on Natural Language Processing (IJCNLP), pages 356–364, Nagoya, Japan. Asian Federation of Natural Lan… [find](https://scholar.google.com/scholar?q=Timothy+Baldwin%2C+Paul+Cook%2C+Marco+Lui%2C+Andrew+MacKinlay%2C+and+Li+Wang%2E+2013%2E+How+noisy+social+media+text%2C+how+diffrnt+social+media+sources%3F+In+Proceedings+of+the+Sixth+International+Joint+Conference+on) [our card](https://md.research.elamur.ai/free/en/papers/baldwin-2013-how-noisy-social-media-text/)
4. Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. COMETA: A corpus for medical entity linking in the social media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3122–3137, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Marco+Basaldella%2C+Fangyu+Liu%2C+Ehsan+Shareghi%2C+and+Nigel+Collier%2E+2020%2E+COMETA%3A+A+corpus+for+medical+entity+linking+in+the+social+media%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Na)
5. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620,… [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Kyle+Lo%2C+and+Arman+Cohan%2E+2019%2E+SciB%2DERT%3A+A+pretrained+language+model+for+scientific+text%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the)
6. Olivier Bodenreider. 2004. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Research, 32:D267–D270. [find](https://scholar.google.com/scholar?q=Olivier+Bodenreider%2E+2004%2E+The+unified+medical+language+system+%28UMLS%29%3A+integrating+biomedical+terminology%2E+Nucleic+Acids+Research%2C+32%3AD267%E2%80%93D270%2E)
7. Allan Peter Davis, Cynthia J Grondin, Robin J Johnson, Daniela Sciaky, Roy McMorran, Jolene Wiegers, Thomas C Wiegers, and Carolyn J Mattingly. 2019. The comparative toxicogenomics database: update 2019. Nucleic Acids Research, 47:D948–D954. [find](https://scholar.google.com/scholar?q=Allan+Peter+Davis%2C+Cynthia+J+Grondin%2C+Robin+J+Johnson%2C+Daniela+Sciaky%2C+Roy+McMorran%2C+Jolene+Wiegers%2C+Thomas+C+Wiegers%2C+and+Carolyn+J+Mattingly%2E+2019%2E+The+comparative+toxicogenomics+database%3A+update+20)
8. Allan Peter Davis, Thomas C Wiegers, Michael C Rosenstein, and Carolyn J Mattingly. 2012. MEDIC: a practical disease vocabulary used at the comparative toxicogenomics database. Database. [find](https://scholar.google.com/scholar?q=Allan+Peter+Davis%2C+Thomas+C+Wiegers%2C+Michael+C+Rosenstein%2C+and+Carolyn+J+Mattingly%2E+2012%2E+MEDIC%3A+a+practical+disease+vocabulary+used+at+the+comparative+toxicogenomics+database%2E+Database%2E)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
10. Jennifer D’Souza and Vincent Ng. 2015. Sieve-based entity linking for the biomedical domain. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP) (Volume 2: Short Papers), pag… [find](https://scholar.google.com/scholar?q=Jennifer+D%E2%80%99Souza+and+Vincent+Ng%2E+2015%2E+Sieve%2Dbased+entity+linking+for+the+biomedical+domain%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+7th+Inte)
11. Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, and Russ R Salakhutdinov. 2005. Neighbourhood components analysis. In Advances in Neural Information Processing Systems, pages 513–520. [find](https://scholar.google.com/scholar?q=Jacob+Goldberger%2C+Geoffrey+E+Hinton%2C+Sam+T+Roweis%2C+and+Russ+R+Salakhutdinov%2E+2005%2E+Neighbourhood+components+analysis%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+513%E2%80%93520%2E)
12. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing. arXiv:2007.15779. [find](https://scholar.google.com/scholar?q=Yu+Gu%2C+Robert+Tinn%2C+Hao+Cheng%2C+Michael+Lucas%2C+Naoto+Usuyama%2C+Xiaodong+Liu%2C+Tristan+Naumann%2C+Jianfeng+Gao%2C+and+Hoifung+Poon%2E+2020%2E+Domain%2Dspecific+language+model+pretraining+for+biomedical+natural+lang) [our card](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/)
13. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Haoqi+Fan%2C+Yuxin+Wu%2C+Saining+Xie%2C+and+Ross+Girshick%2E+2020%2E+Momentum+contrast+for+unsupervised+visual+representation+learning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+a) [our card](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/)
14. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Lon… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
15. Zongcheng Ji, Qiang Wei, and Hua Xu. 2020. BERT-based ranking for biomedical entity normalization. AMIA Summits on Translational Science Proceedings, 2020:269. [find](https://scholar.google.com/scholar?q=Zongcheng+Ji%2C+Qiang+Wei%2C+and+Hua+Xu%2E+2020%2E+BERT%2Dbased+ranking+for+biomedical+entity+normalization%2E+AMIA+Summits+on+Translational+Science+Proceedings%2C+2020%3A269%2E)
16. Donghyeon Kim, Jinhyuk Lee, Chan Ho So, Hwisang Jeon, Minbyul Jeong, Yonghwa Choi, Wonjin Yoon, Mujeen Sung, , and Jaewoo Kang. 2019. A neural named entity recognition and multi-type normalization tool for biomedical text mining. IEEE Access, 7:73729–73740. [find](https://scholar.google.com/scholar?q=Donghyeon+Kim%2C+Jinhyuk+Lee%2C+Chan+Ho+So%2C+Hwisang+Jeon%2C+Minbyul+Jeong%2C+Yonghwa+Choi%2C+Wonjin+Yoon%2C+Mujeen+Sung%2C+%2C+and+Jaewoo+Kang%2E+2019%2E+A+neural+named+entity+recognition+and+multi%2Dtype+normalization+too)
17. Robert Leaman and Zhiyong Lu. 2016. TaggerOne: joint named entity recognition and normalization with semi-markov models. Bioinformatics, 32:2839–2846. [find](https://scholar.google.com/scholar?q=Robert+Leaman+and+Zhiyong+Lu%2E+2016%2E+TaggerOne%3A+joint+named+entity+recognition+and+normalization+with+semi%2Dmarkov+models%2E+Bioinformatics%2C+32%3A2839%E2%80%932846%2E)
18. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. 36(4):1234–1240. [find](https://scholar.google.com/scholar?q=Jinhyuk+Lee%2C+Wonjin+Yoon%2C+Sungdong+Kim%2C+Donghyeon+Kim%2C+Sunkyu+Kim%2C+Chan+Ho+So%2C+and+Jaewoo+Kang%2E+2020%2E+BioBERT%3A+a+pre%2Dtrained+biomedical+language+representation+model+for+biomedical+text+mining%2E+36%284%29%3A) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
19. Bioinformatics, Sunwon Lee, Donghyeon Kim, Kyubum Lee, Jaehoon Choi, Seongsoon Kim, Minji Jeon, Sangrak Lim, Donghee Choi, Sunkyu Kim, Aik-Choon Tan, et al. 2016. BEST: next-generation biomedical entity search tool for knowledge discovery from biomedical literature. PloS one, 11:e0164680. [find](https://scholar.google.com/scholar?q=Bioinformatics%2C+Sunwon+Lee%2C+Donghyeon+Kim%2C+Kyubum+Lee%2C+Jaehoon+Choi%2C+Seongsoon+Kim%2C+Minji+Jeon%2C+Sangrak+Lim%2C+Donghee+Choi%2C+Sunkyu+Kim%2C+Aik%2DChoon+Tan%2C+et+al%2E+2016%2E+BEST%3A+next%2Dgeneration+biomedical+enti)
20. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. [find](https://scholar.google.com/scholar?q=Jiao+Li%2C+Yueping+Sun%2C+Robin+J+Johnson%2C+Daniela+Sci%2Daky%2C+Chih%2DHsuan+Wei%2C+Robert+Leaman%2C+Allan+Peter+Davis%2C+Carolyn+J+Mattingly%2C+Thomas+C+Wiegers%2C+and+Zhiyong+Lu%2E+2016%2E+BioCreative+V+CDR+task+corpus%3A+a+)
21. Nut Limsopatham and Nigel Collier. 2015. Adapting phrase-based machine translation to normalise medical terms in social media messages. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1675– 1680, Lisbon, Portugal. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Nut+Limsopatham+and+Nigel+Collier%2E+2015%2E+Adapting+phrase%2Dbased+machine+translation+to+normalise+medical+terms+in+social+media+messages%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Na)
22. Nut Limsopatham and Nigel Collier. 2016. Normalising medical concepts in social media texts by learning semantic representation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1014–1023. [find](https://scholar.google.com/scholar?q=Nut+Limsopatham+and+Nigel+Collier%2E+2016%2E+Normalising+medical+concepts+in+social+media+texts+by+learning+semantic+representation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Comput)
23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
24. Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing, pages 58–65. [find](https://scholar.google.com/scholar?q=Yifan+Peng%2C+Shankai+Yan%2C+and+Zhiyong+Lu%2E+2019%2E+Transfer+learning+in+biomedical+natural+language+processing%3A+An+evaluation+of+bert+and+elmo+on+ten+benchmarking+datasets%2E+In+Proceedings+of+the+2019+Work) [our card](https://md.research.elamur.ai/free/en/papers/peng-2019-transfer-learning-biomedical-natural-language/)
25. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+2020%2E+MAD%2DX%3A+An+Adapter%2DBased+Framework+for+Multi%2DTask+Cross%2DLingual+Transfer%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Meth) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/)
26. Minh C Phan, Aixin Sun, and Yi Tay. 2019. Robust representation learning of biomedical names. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3275– 3285. [find](https://scholar.google.com/scholar?q=Minh+C+Phan%2C+Aixin+Sun%2C+and+Yi+Tay%2E+2019%2E+Robust+representation+learning+of+biomedical+names%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+3275%E2%80%93+32)
27. Kirk Roberts, Matthew S Simpson, Ellen M Voorhees, and William R Hersh. 2015. Overview of the trec 2015 clinical decision support track. In TREC. [find](https://scholar.google.com/scholar?q=Kirk+Roberts%2C+Matthew+S+Simpson%2C+Ellen+M+Voorhees%2C+and+William+R+Hersh%2E+2015%2E+Overview+of+the+trec+2015+clinical+decision+support+track%2E+In+TREC%2E)
28. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823. [find](https://scholar.google.com/scholar?q=Florian+Schroff%2C+Dmitry+Kalenichenko%2C+and+James+Philbin%2E+2015%2E+Facenet%3A+A+unified+embedding+for+face+recognition+and+clustering%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Re) [our card](https://md.research.elamur.ai/free/en/papers/schroff-2015-facenet-unified-embedding-face-recognition/)
29. Elliot Schumacher, Andriy Mulyar, and Mark Dredze. 2020. Clinical concept linking with contextualized neural representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8585–8592. [find](https://scholar.google.com/scholar?q=Elliot+Schumacher%2C+Andriy+Mulyar%2C+and+Mark+Dredze%2E+2020%2E+Clinical+concept+linking+with+contextualized+neural+representations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computati)
30. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2DSNE%2E+Journal+of+machine+learning+research%2C+9%28Nov%29%3A2579%E2%80%932605%2E)
31. Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020. BioMegatron: Larger biomedical domain language model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4700–4706, Online. Assoc… [find](https://scholar.google.com/scholar?q=Hoo%2DChang+Shin%2C+Yang+Zhang%2C+Evelina+Bakhturina%2C+Raul+Puri%2C+Mostofa+Patwary%2C+Mohammad+Shoeybi%2C+and+Raghav+Mani%2E+2020%2E+BioMegatron%3A+Larger+biomedical+domain+language+model%2E+In+Proceedings+of+the+2020+Co)
32. George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen Chen, and Alex Wong. 2020. Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus. arXiv preprint arXiv:2010.10391. [find](https://scholar.google.com/scholar?q=George+Michalopoulos%2C+Yuanxin+Wang%2C+Hussam+Kaka%2C+Helen+Chen%2C+and+Alex+Wong%2E+2020%2E+Umlsbert%3A+Clinical+domain+knowledge+augmentation+of+contextual+embeddings+using+the+unified+medical+language+system+me)
33. Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. 2020. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6398–6407. [find](https://scholar.google.com/scholar?q=Yifan+Sun%2C+Changmao+Cheng%2C+Yuhan+Zhang%2C+Chi+Zhang%2C+Liang+Zheng%2C+Zhongdao+Wang%2C+and+Yichen+Wei%2E+2020%2E+Circle+loss%3A+A+unified+perspective+of+pair+similarity+optimization%2E+In+Proceedings+of+the+IEEE%2FCVF+)
34. Sunil Mohan and Donghui Li. 2018. MedMentions: A large biomedical corpus annotated with UMLS concepts. In Automated Knowledge Base Construction. [find](https://scholar.google.com/scholar?q=Sunil+Mohan+and+Donghui+Li%2E+2018%2E+MedMentions%3A+A+large+biomedical+corpus+annotated+with+UMLS+concepts%2E+In+Automated+Knowledge+Base+Construction%2E)
35. Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020. Biomedical entity representations with synonym marginalization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3641–3650, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mujeen+Sung%2C+Hwisang+Jeon%2C+Jinhyuk+Lee%2C+and+Jaewoo+Kang%2E+2020%2E+Biomedical+entity+representations+with+synonym+marginalization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computat)
36. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2018%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
37. Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4004–4012. [find](https://scholar.google.com/scholar?q=Hyun+Oh+Song%2C+Yu+Xiang%2C+Stefanie+Jegelka%2C+and+Silvio+Savarese%2E+2016%2E+Deep+metric+learning+via+lifted+structured+feature+embedding%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+)
38. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. [find](https://scholar.google.com/scholar?q=Aaron+van+den+Oord%2C+Yazhe+Li%2C+and+Oriol+Vinyals%2E+2018%2E+Representation+learning+with+contrastive+predictive+coding%2E+arXiv+preprint+arXiv%3A1807%2E03748%2E) [our card](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/)
39. Elena Tutubalina, Artur Kadurin, and Zulfat Miftahutdinov. 2020. Fair evaluation in concept normalization: a large-scale comparative analysis for bert-based models. In Proceedings of the 28th International Conference on Computational Linguistics (COLING). [find](https://scholar.google.com/scholar?q=Elena+Tutubalina%2C+Artur+Kadurin%2C+and+Zulfat+Miftahutdinov%2E+2020%2E+Fair+evaluation+in+concept+normalization%3A+a+large%2Dscale+comparative+analysis+for+bert%2Dbased+models%2E+In+Proceedings+of+the+28th+Internat)
40. Elena Tutubalina, Zulfat Miftahutdinov, Sergey Nikolenko, and Valentin Malykh. 2018. Medical concept normalization in social media posts with recurrent neural networks. Journal of Biomedical Informatics, 84:93–102. [find](https://scholar.google.com/scholar?q=Elena+Tutubalina%2C+Zulfat+Miftahutdinov%2C+Sergey+Nikolenko%2C+and+Valentin+Malykh%2E+2018%2E+Medical+concept+normalization+in+social+media+posts+with+recurrent+neural+networks%2E+Journal+of+Biomedical+Informati)
41. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computationa… [find](https://scholar.google.com/scholar?q=Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Robert+Litschko%2C+Goran+Glava%C5%A1%2C+and+Anna+Korhonen%2E+2020%2E+Probing+pretrained+language+models+for+lexical+semantics%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+M) [our card](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/)
42. Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. 2019. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5022– 5030. [find](https://scholar.google.com/scholar?q=Xun+Wang%2C+Xintong+Han%2C+Weilin+Huang%2C+Dengke+Dong%2C+and+Matthew+R+Scott%2E+2019%2E+Multi%2Dsimilarity+loss+with+general+pair+weighting+for+deep+metric+learning%2E+In+Proceedings+of+the+IEEE+Conference+on+Comput)
43. Yanshan Wang, Sijia Liu, Naveed Afzal, Majid Rastegar-Mojarad, Liwei Wang, Feichen Shen, Paul Kingsbury, and Hongfang Liu. 2018. A comparison of word embeddings for the biomedical natural language processing. Journal of Biomedical Informatics, 87:12–20. [find](https://scholar.google.com/scholar?q=Yanshan+Wang%2C+Sijia+Liu%2C+Naveed+Afzal%2C+Majid+Rastegar%2DMojarad%2C+Liwei+Wang%2C+Feichen+Shen%2C+Paul+Kingsbury%2C+and+Hongfang+Liu%2E+2018%2E+A+comparison+of+word+embeddings+for+the+biomedical+natural+language+pro)
44. Dustin Wright, Yannis Katsis, Raghav Mehta, and Chun-Nan Hsu. 2019. Normco: Deep disease normalization for biomedical knowledge base construction. In Automated Knowledge Base Construction. [find](https://scholar.google.com/scholar?q=Dustin+Wright%2C+Yannis+Katsis%2C+Raghav+Mehta%2C+and+Chun%2DNan+Hsu%2E+2019%2E+Normco%3A+Deep+disease+normalization+for+biomedical+knowledge+base+construction%2E+In+Automated+Knowledge+Base+Construction%2E)
45. Dongfang Xu, Zeyu Zhang, and Steven Bethard. 2020. A generate-and-rank framework with semantic type regularization for biomedical concept normalization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8452–8464. [find](https://scholar.google.com/scholar?q=Dongfang+Xu%2C+Zeyu+Zhang%2C+and+Steven+Bethard%2E+2020%2E+A+generate%2Dand%2Drank+framework+with+semantic+type+regularization+for+biomedical+concept+normalization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+th)
46. The chemical mentions are mapped into the Comparative Toxicogenomics Database (CTD) (Davis et al., 2019) chemical dictionary. We denote the disease and chemical mention sets as “BC5CDR-d” and “BC5CDR-c” respectively. For NCBI and BC5CDR we use the same data and evaluation protocol by Sung et al. (20… [find](https://scholar.google.com/scholar?q=The+chemical+mentions+are+mapped+into+the+Comparative+Toxicogenomics+Database+%28CTD%29+%28Davis+et+al%2E%2C+2019%29+chemical+dictionary%2E+We+denote+the+disease+and+chemical+mention+sets+as+%E2%80%9CBC5CDR%2Dd%E2%80%9D+and+%E2%80%9CBC5CDR%2D)
47. MedMentions (Mohan and Li, 2018) is a very-large-scale entity linking dataset containing over 4,000 abstracts and over 350,000 mentions linked to UMLS 2017AA. According to Mohan and Li (2018), training T AGGER O NE (Leaman and Lu, 2016), a very popular MEL system, on a subset of MedMentions require… [find](https://scholar.google.com/scholar?q=MedMentions+%28Mohan+and+Li%2C+2018%29+is+a+very%2Dlarge%2Dscale+entity+linking+dataset+containing+over+4%2C000+abstracts+and+over+350%2C000+mentions+linked+to+UMLS+2017AA%2E+According+to+Mohan+and+Li+%282018%29%2C+trainin)
48. A.2 Social-Media Language Datasets AskAPatient (Limsopatham and Collier, 2016) includes 17,324 adverse drug reaction (ADR) annotations collected from askapatient.com blog posts. The mentions are mapped to 1,036 medical concepts grounded onto SNOMED-CT (Donnelly, 2006) and AMT (the Australian Medicin… [find](https://scholar.google.com/scholar?q=A%2E2+Social%2DMedia+Language+Datasets+AskAPatient+%28Limsopatham+and+Collier%2C+2016%29+includes+17%2C324+adverse+drug+reaction+%28ADR%29+annotations+collected+from+askapatient%2Ecom+blog+posts%2E+The+mentions+are+mappe)
49. We divide our experimental datasets into two categories (1) scientific language datasests where the data is extracted from scientific papers and (2) social media language datasets where the data is coming from social media forums like Reddit.com. For an overview of the key statistics, see Tab. 3. [find](https://scholar.google.com/scholar?q=We+divide+our+experimental+datasets+into+two+categories+%281%29+scientific+language+datasests+where+the+data+is+extracted+from+scientific+papers+and+%282%29+social+media+language+datasets+where+the+data+is+co)
50. COMETA (Basaldella et al., 2020) is a recently released large-scale MEL dataset that specifically focuses on MEL in the social media domain, containing around 20k medical mentions extracted from health-related discussions on reddit.com. Mentions are mapped to SNOMED-CT. We use the “stratified (gener… [find](https://scholar.google.com/scholar?q=COMETA+%28Basaldella+et+al%2E%2C+2020%29+is+a+recently+released+large%2Dscale+MEL+dataset+that+specifically+focuses+on+MEL+in+the+social+media+domain%2C+containing+around+20k+medical+mentions+extracted+from+healt)