---
title: "Fast, Effective, and Self-Supervised — paper by Liu et al. (2021)"
description: "Liu et al. propose Mirror-BERT, a fully self-supervised fine-tuning recipe that needs no external… 79 references, cited by 63 papers here."
authors: ["Fangyu Liu", "Ivan Vulić", "Anna Korhonen", "Nigel Collier"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.109/"
doi: "10.18653/v1/2021.emnlp-main.109"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2021-fast-effective-self-supervised-transforming/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2021-fast-effective-self-supervised-transforming/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

# Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

**Authors:** [Fangyu Liu](https://md.research.elamur.ai/free/en/authors/fangyu-liu/), [Ivan Vulić](https://md.research.elamur.ai/free/en/authors/ivan-vulic/), [Anna Korhonen](https://md.research.elamur.ai/free/en/authors/anna-korhonen/) and [Nigel Collier](https://md.research.elamur.ai/free/en/authors/nigel-collier/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.109/) · [PDF](https://aclanthology.org/2021.emnlp-main.109.pdf) · [DOI 10.18653/v1/2021.emnlp-main.109](https://doi.org/10.18653/v1/2021.emnlp-main.109) · [OpenAlex](https://openalex.org/W3213730158)

**Citations:** 84 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/liu-2021-fast-effective-self-supervised-transforming/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Pretrained masked language models such as BERT and RoBERTa hold rich semantic knowledge. Used as-is, however, they give weak word and sentence embeddings for similarity tasks. Earlier fixes such as Sentence-BERT and SapBERT depend on human-labelled resources like NLI data or UMLS synonyms. Liu et al. propose Mirror-BERT, a fully self-supervised fine-tuning recipe that needs no external annotation.

Each string from a small unlabelled set is copied to form a positive pair. The two copies are made slightly different in two ways: random span masking of the input (for phrases and sentences) and the dropout layers already inside the network. An InfoNCE contrastive loss then pulls each pair together and treats the other strings in the batch as negatives. The whole conversion runs in under a minute on two 2080Ti GPUs.

Mirror-BERT clearly beats off-the-shelf MLMs on word similarity, biomedical entity linking and sentence-level tasks, across several languages and domains. On STS and QNLI it even matches supervised Sentence-BERT. Universal encoders can therefore be built cheaply where annotated data is scarce.

## Key points
- Lexical evaluation uses Multi-SimLex (Spearman's ρ) and five biomedical entity linking sets: NCBI-disease, BC5CDR-disease, BC5CDR-chemical, AskAPatient and COMETA (accuracy @1/@5).
- Masking and dropout work synergistically. Dropout alone also works for single-token words, where span masking is not applicable.
- Fine-tuning on random strings of letters and digits still lifts EN Multi-SimLex from ρ = 0.267 to 0.481. Frequent, in-domain text works best, and a domain shift lowers QNLI AUC (.674 → .665).
- About 10k unlabelled items per task suffice, with a reported transformation time of 20–30 seconds.
- Code and models: github.com/cambridgeltl/mirror-bert. The concurrent SimCSE is described as a special case of this framework.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Fangyu Liu, Ivan Vulić, Anna Korhonen and Nigel Collier. 2021. Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2021fast,
  title = {{Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders}},
  author = {Fangyu Liu, Ivan Vulić, Anna Korhonen and Nigel Collier},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.109},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.109}
}
```

- [← previous Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary](https://md.research.elamur.ai/free/en/papers/deutsch-2021-question-answering-as-automatic-metric/)
- [next → Introducing CAD: the Contextual Abuse Dataset](https://md.research.elamur.ai/free/en/papers/vidgen-2021-introducing-cad-contextual-abuse-dataset/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

79 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. [SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretabi…](https://md.research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/)
2. Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. COMETA: A corpus for medical entity linking in the social media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3122–3137, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Marco+Basaldella%2C+Fangyu+Liu%2C+Ehsan+Shareghi%2C+and+Nigel+Collier%2E+2020%2E+COMETA%3A+A+corpus+for+medical+entity+linking+in+the+social+media%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Na)
3. Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2020. BitFit: Simple parameter-efficient fine-tuningfor transformer-based masked language-model. Technical Report. [find](https://scholar.google.com/scholar?q=Elad+Ben%2DZaken%2C+Shauli+Ravfogel%2C+and+Yoav+Goldberg%2E+2020%2E+BitFit%3A+Simple+parameter%2Defficient+fine%2Dtuningfor+transformer%2Dbased+masked+language%2Dmodel%2E+Technical+Report%2E)
4. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. [SemEval-2014 task 10: Multilingual semantic textual similarity](https://md.research.elamur.ai/free/en/papers/agirre-2014-semeval-2014-task-10-multilingual/). In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2…
5. Olivier Bodenreider. 2004. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 32(suppl_1):D267– D270. [find](https://scholar.google.com/scholar?q=Olivier+Bodenreider%2E+2004%2E+The+unified+medical+language+system+%28umls%29%3A+integrating+biomedical+terminology%2E+Nucleic+acids+research%2C+32%28suppl%5F1%29%3AD267%E2%80%93+D270%2E)
6. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. [SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval…
7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
8. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. [SemEval-2012 task 6: A pilot on semantic textual similarity](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/). In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceed…
9. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. [*SEM 2013 shared task: Semantic textual similarity](https://md.research.elamur.ai/free/en/papers/agirre-2013-sem-2013-shared-task-semantic/). In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarit…
10. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. [A latent variable model approach to PMI-based word embeddings](https://md.research.elamur.ai/free/en/papers/arora-2016-latent-variable-model-approach-pmi/). Transactions of the Association for Computational Linguistics, 4:385–399.
11. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://md.research.elamur.ai/free/en/papers/artetxe-2018-vecmap-unsupervised/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia…
12. Xavier Bouthillier, Kishore Konda, Pascal Vincent, and Roland Memisevic. 2015. Dropout as data augmentation. ArXiv preprint, abs/1506.08700. — [arXiv:1506.08700](https://arxiv.org/abs/1506.08700) [find](https://scholar.google.com/scholar?q=Xavier+Bouthillier%2C+Kishore+Konda%2C+Pascal+Vincent%2C+and+Roland+Memisevic%2E+2015%2E+Dropout+as+data+augmentation%2E+ArXiv+preprint%2C+abs%2F1506%2E08700%2E+%E2%80%94+%5BarXiv%3A1506%2E08700%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1506%2E08700%29)
13. Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [find](https://scholar.google.com/scholar?q=Xingyu+Cai%2C+Jiaji+Huang%2C+Yuchen+Bian%2C+and+Kenneth+Church%2E+2021%2E+Isotropy+in+the+contextual+embedding+space%3A+Clusters+and+manifolds%2E+In+9th+International+Conference+on+Learning+Representations%2C+ICLR+20)
14. Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [find](https://scholar.google.com/scholar?q=Fredrik+Carlsson%2C+Amaru+Cuba+Gyllensten%2C+Evangelia+Gogoulou%2C+Erik+Ylip%C3%A4%C3%A4+Hellqvist%2C+and+Magnus+Sahlgren%2E+2021%2E+Semantic+re%2Dtuning+with+contrastive+tension%2E+In+9th+International+Conference+on+Learning+)
15. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. [SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.…
16. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. [Universal sentence encoder for English](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder-english/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua…
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
18. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. [find](https://scholar.google.com/scholar?q=Rezarta+Islamaj+Dog%CC%86an%2C+Robert+Leaman%2C+and+Zhiyong+Lu%2E+2014%2E+Ncbi+disease+corpus%3A+a+resource+for+disease+name+recognition+and+concept+normalization%2E+Journal+of+biomedical+informatics%2C+47%3A1%E2%80%9310%2E)
19. Maud Ehrmann, Francesco Cecconi, Daniele Vannella, John Philip McCrae, Philipp Cimiano, and Roberto Navigli. 2014. Representing multilingual data as linked data: the case of BabelNet 2.0. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 401–4… [find](https://scholar.google.com/scholar?q=Maud+Ehrmann%2C+Francesco+Cecconi%2C+Daniele+Vannella%2C+John+Philip+McCrae%2C+Philipp+Cimiano%2C+and+Roberto+Navigli%2E+2014%2E+Representing+multilingual+data+as+linked+data%3A+the+case+of+BabelNet+2%2E0%2E+In+Proceedin)
20. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+comparing+the+geometry+of+BERT%2C+ELMo%2C+and+GPT%2D2+embeddings%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+i)
21. Christiane Fellbaum. 1998. WordNet. MIT Press. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998%2E+WordNet%2E+MIT+Press%2E)
22. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. [Language-agnostic bert sentence embedding](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/). ArXiv preprint, abs/2007.01852.
23. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. [SimCSE: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). ArXiv preprint, abs/2104.08821.
24. John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. [DeCLUTR: Deep contrastive learning for unsupervised textual representations](https://md.research.elamur.ai/free/en/papers/giorgi-2021-declutr-deep-contrastive-learning-unsupervised/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin…
25. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ArXiv preprint, abs/2007.15779.
26. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. [find](https://scholar.google.com/scholar?q=Zellig+S+Harris%2E+1954%2E+Distributional+structure%2E+Word%2C+10%282%2D3%29%3A146%E2%80%93162%2E)
27. Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019. Training neural response selection for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Associat… [find](https://scholar.google.com/scholar?q=Matthew+Henderson%2C+Ivan+Vulic%CC%81%2C+Daniela+Gerz%2C+I%C3%B1igo+Casanueva%2C+Pawe%C5%82+Budzianowski%2C+Sam+Coope%2C+Georgios+Spithourakis%2C+Tsung%2DHsien+Wen%2C+Nikola+Mrk%C5%A1ic%CC%81%2C+and+Pei%2DHao+Su%2E+2019%2E+Training+neural+response+sel)
28. Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. 2020. Augmix: A simple data processing method to improve robustness and uncertainty. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.… [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Norman+Mu%2C+Ekin+Dogus+Cubuk%2C+Barret+Zoph%2C+Justin+Gilmer%2C+and+Balaji+Lakshminarayanan%2E+2020%2E+Augmix%3A+A+simple+data+processing+method+to+improve+robustness+and+uncertainty%2E+In+8th+Interna)
29. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. [Learning distributed representations of sentences from unlabelled data](https://md.research.elamur.ai/free/en/papers/hill-2016-learning-distributed-representations-sentences-unlabelled/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1367–1377, San Diego, C…
30. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. [SimLex-999: Evaluating semantic models with (genuine) similarity estimation](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/). Computational Linguistics, 41(4):665–695.
31. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview… [find](https://scholar.google.com/scholar?q=Samuel+Humeau%2C+Kurt+Shuster%2C+Marie%2DAnne+Lachaux%2C+and+Jason+Weston%2E+2020%2E+Poly%2Dencoders%3A+Architectures+and+pre%2Dtraining+strategies+for+fast+and+accurate+multi%2Dsentence+scoring%2E+In+8th+International+Con)
32. Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 710–7… [find](https://scholar.google.com/scholar?q=Goran+Glava%C5%A1%2C+Robert+Litschko%2C+Sebastian+Ruder%2C+and+Ivan+Vulic%CC%81%2E+2019%2E+How+to+%28properly%29+evaluate+cross%2Dlingual+word+embeddings%3A+On+strong+baselines%2C+comparative+analyses%2C+and+some+misconceptions%2E+In+)
33. Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. [Self-guided contrastive learning for BERT sentence representations](https://md.research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long P…
34. Goran Glavaš and Ivan Vulić. 2021. Is supervised syntactic parsing beneficial for language understanding tasks? an empirical investigation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3090–3104, Online. Associati… [find](https://scholar.google.com/scholar?q=Goran+Glava%C5%A1+and+Ivan+Vulic%CC%81%2E+2021%2E+Is+supervised+syntactic+parsing+beneficial+for+language+understanding+tasks%3F+an+empirical+investigation%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapt)
35. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Queb… [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+Salakhutdinov%2C+Richard+S%2E+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+Advances+in+Neural+Information+Processing+Systems+28%3A+)
36. Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. [Revealing the dark secrets of BERT](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365–437…
37. Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Alexis+Conneau%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herv%C3%A9+J%C3%A9gou%2E+2018%2E+Word+translation+without+parallel+data%2E+In+6th+International+Conference+on+Learning+Representations%2C+ICL)
38. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. [On the sentence embeddings from pre-trained language models](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119–9130, Online. Association for Computational Linguis…
39. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. [find](https://scholar.google.com/scholar?q=Jiao+Li%2C+Yueping+Sun%2C+Robin+J+Johnson%2C+Daniela+Sci%2Daky%2C+Chih%2DHsuan+Wei%2C+Robert+Leaman%2C+Allan+Peter+Davis%2C+Carolyn+J+Mattingly%2C+Thomas+C+Wiegers%2C+and+Zhiyong+Lu%2E+2016%2E+Biocreative+v+cdr+task+corpus%3A+a+)
40. Nut Limsopatham and Nigel Collier. 2016. Normalising medical concepts in social media texts by learning semantic representation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1014–1023, Berlin, Germany. Association for Compu… [find](https://scholar.google.com/scholar?q=Nut+Limsopatham+and+Nigel+Collier%2E+2016%2E+Normalising+medical+concepts+in+social+media+texts+by+learning+semantic+representation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Comput)
41. Robert Litschko, Ivan Vulić, Simone Paolo Ponzetto, and Goran Glavaš. 2021. Evaluating multilingual text encoders for unsupervised cross-lingual retrieval. In Proceedings of 43rd European Conference on Information Retrieval (ECIR 2021), pages 342–358. [find](https://scholar.google.com/scholar?q=Robert+Litschko%2C+Ivan+Vulic%CC%81%2C+Simone+Paolo+Ponzetto%2C+and+Goran+Glava%C5%A1%2E+2021%2E+Evaluating+multilingual+text+encoders+for+unsupervised+cross%2Dlingual+retrieval%2E+In+Proceedings+of+43rd+European+Conference+)
42. Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021. [Self-alignment pretraining for biomedical entity representations](https://md.research.elamur.ai/free/en/papers/liu-2021-self-alignment-pretraining-biomedical-entity/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa…
43. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+A)
44. Lajanugen Logeswaran and Honglak Lee. 2018. [An efficient framework for learning sentence representations](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/). In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
45. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+OpenReview%2En)
46. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. [A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages…
47. Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021. [Coco-lm: Correcting and contrasting text sequences for language model pretraining](https://md.research.elamur.ai/free/en/papers/meng-2021-coco-lm-correcting-contrasting-text/). ArXiv preprint, abs/2102.08473.
48. Paul Michel, Omer Levy, and Graham Neubig. 2019. [Are sixteen heads really better than one?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/) In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14014– 14024.
49. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). ArXiv preprint, abs/1301.3781.
50. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. [Advances in pre-training distributed word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resource…
51. Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceeding…
52. Jiaqi Mu and Pramod Viswanath. 2018. All-but-the-top: Simple and effective postprocessing for word representations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. [find](https://scholar.google.com/scholar?q=Jiaqi+Mu+and+Pramod+Viswanath%2E+2018%2E+All%2Dbut%2Dthe%2Dtop%3A+Simple+and+effective+postprocessing+for+word+representations%2E+In+6th+International+Conference+on+Learning+Representations%2C+ICLR+2018%2C+Vancouver%2C+B)
53. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). ArXiv preprint, abs/1807.03748.
54. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. [PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification](https://md.research.elamur.ai/free/en/papers/pavlick-2015-ppdb-2-0-better-paraphrase/). In Proceedings of the 53rd Annual Meeting of the Association for Computational…
55. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
56. Mohammad Taher Pilehvar, Dimitri Kartsaklis, Victor Prokhorov, and Nigel Collier. 2018. Card-660: Cambridge rare word dataset - a reliable benchmark for infrequent word representation models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1391– 1401,… [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar%2C+Dimitri+Kartsaklis%2C+Victor+Prokhorov%2C+and+Nigel+Collier%2E+2018%2E+Card%2D660%3A+Cambridge+rare+word+dataset+%2D+a+reliable+benchmark+for+infrequent+word+representation+models%2E+In+Proce)
57. Sara Rajaee and Mohammad Taher Pilehvar. 2021. A cluster-based approach for improving isotropy in contextual embedding space. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… [find](https://scholar.google.com/scholar?q=Sara+Rajaee+and+Mohammad+Taher+Pilehvar%2E+2021%2E+A+cluster%2Dbased+approach+for+improving+isotropy+in+contextual+embedding+space%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computati)
58. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
59. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
60. Nils Rethmeier and Isabelle Augenstein. 2021. A primer on contrastive pretraining in language processing: Methods, lessons learned and perspectives. ArXiv preprint, abs/2102.12982. — [arXiv:2102.12982](https://arxiv.org/abs/2102.12982) [find](https://scholar.google.com/scholar?q=Nils+Rethmeier+and+Isabelle+Augenstein%2E+2021%2E+A+primer+on+contrastive+pretraining+in+language+processing%3A+Methods%2C+lessons+learned+and+perspectives%2E+ArXiv+preprint%2C+abs%2F2102%2E12982%2E+%E2%80%94+%5BarXiv%3A2102%2E12982)
61. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). Transactions of the Association for Computational Linguistics, 8:842–866.
62. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021. Wiki-Matrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages… [find](https://scholar.google.com/scholar?q=Holger+Schwenk%2C+Vishrav+Chaudhary%2C+Shuo+Sun%2C+Hongyu+Gong%2C+and+Francisco+Guzm%C3%A1n%2E+2021%2E+Wiki%2DMatrix%3A+Mining+135M+parallel+sentences+in+1620+language+pairs+from+Wikipedia%2E+In+Proceedings+of+the+16th+Conf)
63. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+The+journal+of+machine+learning+)
64. Jianlin Su. 2020. Simbert: Integrating retrieval and generation into bert. Technical report. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2E+2020%2E+Simbert%3A+Integrating+retrieval+and+generation+into+bert%2E+Technical+report%2E)
65. Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. ArXiv preprint, abs/2103.15316. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Jiarun+Cao%2C+Weijie+Liu%2C+and+Yangyiwen+Ou%2E+2021%2E+Whitening+sentence+representations+for+better+semantics+and+faster+retrieval%2E+ArXiv+preprint%2C+abs%2F2103%2E15316%2E)
66. Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021. [Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks](https://md.research.elamur.ai/free/en/papers/thakur-2021-augmented-sbert-data-augmentation-method/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:…
67. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I…
68. Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, and Anna Korhonen. 2020a. Multi-SimLex: A large-scale evaluation of multilingual and crosslingual lexical semantic similarity. Computational Lingui… [find](https://scholar.google.com/scholar?q=Ivan+Vulic%CC%81%2C+Simon+Baker%2C+Edoardo+Maria+Ponti%2C+Ulla+Petti%2C+Ira+Leviant%2C+Kelly+Wing%2C+Olga+Majewska%2C+Eden+Bar%2C+Matt+Malone%2C+Thierry+Poibeau%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2020a%2E+Multi%2DSimLex%3A+A+large)
69. Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2021. LexFit: Lexical fine-tuning of pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V… [find](https://scholar.google.com/scholar?q=Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Anna+Korhonen%2C+and+Goran+Glava%C5%A1%2E+2021%2E+LexFit%3A+Lexical+fine%2Dtuning+of+pretrained+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Com)
70. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020b. [Probing pretrained language models for lexical semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computation…
71. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems 32: Annual Conference on Neural…
72. Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. 2020. Scheduled DropHead: A regularization method for transformer models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1971–1980, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou%2C+Tao+Ge%2C+Furu+Wei%2C+Ming+Zhou%2C+and+Ke+Xu%2E+2020%2E+Scheduled+DropHead%3A+A+regularization+method+for+transformer+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+)
73. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevi…
74. Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learn… [find](https://scholar.google.com/scholar?q=Tongzhou+Wang+and+Phillip+Isola%2E+2020%2E+Understanding+contrastive+representation+learning+through+alignment+and+uniformity+on+the+hypersphere%2E+In+Proceedings+of+the+37th+International+Conference+on+Mac)
75. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
76. Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. ArXiv preprint, abs/2012.15466. [find](https://scholar.google.com/scholar?q=Zhuofeng+Wu%2C+Sinong+Wang%2C+Jiatao+Gu%2C+Madian+Khabsa%2C+Fei+Sun%2C+and+Hao+Ma%2E+2020%2E+Clear%3A+Contrastive+learning+for+sentence+representation%2E+ArXiv+preprint%2C+abs%2F2012%2E15466%2E)
77. Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. [ConSERT: A contrastive framework for self-supervised sentence representation transfer](https://md.research.elamur.ai/free/en/papers/yan-2021-consert-contrastive-framework-self-supervised/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc…
78. Yan Zhang, Ruidan He, Zuozhu Liu, Lidong Bing, and Haizhou Li. 2021. Bootstrapped unsupervised sentence representation learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volu… [find](https://scholar.google.com/scholar?q=Yan+Zhang%2C+Ruidan+He%2C+Zuozhu+Liu%2C+Lidong+Bing%2C+and+Haizhou+Li%2E+2021%2E+Bootstrapped+unsupervised+sentence+representation+learning%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Comput)
79. Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. [An unsupervised sentence embedding method by mutual information maximization](https://md.research.elamur.ai/free/en/papers/zhang-2020-unsupervised-sentence-embedding-method-mutual/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1601–1610, Online. Association for Computation…

*Updated 24 October 2026*