---
title: "Self-Guided Contrastive Learning for BERT — Kim 2021"
description: "Pre-trained encoders like BERT do not produce good sentence vectors out of the box when labeled data are unavailable."
authors: ["Taeuk Kim", "Kang Min Yoo", "Sang-goo Lee"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.197/"
doi: "10.18653/v1/2021.acl-long.197"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Self-Guided Contrastive Learning for BERT Sentence Representations

# Self-Guided Contrastive Learning for BERT Sentence Representations

**Authors:** [Taeuk Kim](https://md.research.elamur.ai/free/en/authors/taeuk-kim/), [Kang Min Yoo](https://md.research.elamur.ai/free/en/authors/kang-min-yoo/), Sang-goo Lee · **Year:** 2021 · **Venue:** Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.197/) · [PDF](https://aclanthology.org/2021.acl-long.197.pdf) · [DOI 10.18653/v1/2021.acl-long.197](https://doi.org/10.18653/v1/2021.acl-long.197) · [OpenAlex](https://openalex.org/W3176047188)

**Citations:** 157 citations (openalex-snapshot · 2026-10-14) · [77](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Pre-trained encoders like BERT do not produce good sentence vectors out of the box when labeled data are unavailable. Using the raw [CLS] output works poorly, and the quality of pooled embeddings changes a great deal depending on the chosen layer and pooling scheme. The authors propose an unsupervised fix based on self-guided contrastive learning. BERT is duplicated into a frozen copy (BERT F) and a trainable copy (BERT T). Sentence-level views are taken from the frozen copy's intermediate layers using max pooling and uniform layer sampling. These views act as positives that the last-layer [CLS] vector of BERT T should move toward, while other sentences in the batch serve as negatives. The method needs only raw sentences and no data augmentation. The authors also redesign the NT-Xent objective from computer vision for this setting, yielding the SG-OPT variant. Across sentence-level tasks, the approach beats mean pooling, WK pooling and flow-based baselines. It is also the fastest at inference, since no post-processing is required, and it holds up well when the training domain differs from the test domain. For practitioners, this makes the ordinary [CLS] vector a usable, low-cost sentence embedding.

## Key points
- In a preliminary STS-B test, BERT-base Spearman correlation (×100) ranged from 16.71 ([CLS], 10th layer) to 63.19 (max pooling, 2nd layer), showing how unstable standard embedding choices are.
- During training, BERT F stays fixed so the guidance signal does not degrade. BERT T is fine-tuned, with Layer 0 excluded, and its final [CLS] vector becomes the sentence embedding.
- On STS-B with a Titan Xp GPU and batch size 16, SG-OPT training took about 455 s (under 8 min). Inference took about 10.5 s, compared with about 13.9 s for mean pooling, about 28.5 s for the flow model and about 197 s for WK pooling.
- Training SG-OPT on mixed NLI data (SNLI/MultiNLI) rather than STS-B text tested robustness to domain shift, and performance stayed relatively stable.
- An exploratory ensemble fed original sentences to BERT T and back-translations to BERT F. It generally improved results further, suggesting that the method can be combined with augmentation-based contrastive techniques.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 74 not yet.

- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/chuang-2022-diffcse-difference-based-contrastive-learning/)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/)
- [Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations](https://md.research.elamur.ai/free/en/papers/long-2022-facilitating-contrastive-learning-discourse-relational/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kim2021self,
  title = {{Self-Guided Contrastive Learning for BERT Sentence Representations}},
  author = {Kim, Taeuk and Yoo, Kang Min and Lee, Sang-goo},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.197},
  url = {https://doi.org/10.18653/v1/2021.acl-long.197}
}
```

- [← previous Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
- [next → Semantic Ambiguity of So-Called "Stative Verbs" in Biblical Hebrew: Beyond Active vs. Stative Binary Opposition for the Qatal and Qatil Forms in the Qal Stem](https://md.research.elamur.ai/free/en/papers/hbo-2021-sato-semantic-ambiguity-so-called-stative/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

46 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2020. Better fine-tuning by reducing representational collapse. arXiv preprint arXiv:2008.03156. — [arXiv:2008.03156](https://arxiv.org/abs/2008.03156) [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Akshat+Shrivastava%2C+Anchit+Gupta%2C+Naman+Goyal%2C+Luke+Zettlemoyer%2C+and+Sonal+Gupta%2E+2020%2E+Better+fine%2Dtuning+by+reducing+representational+collapse%2E+arXiv+preprint+arXiv%3A2008%2E03156%2E+%E2%80%94+%5B)
2. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretab… [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Claire+Cardie%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Weiwei+Guo%2C+In%CC%83igo+Lopez%2DGazpio%2C+Montse+Maritxalar%2C+Rada+Mihalcea%2C+German+Rigau%2C+Larraitz+Uria%2C+and+Janyce+Wieb) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/)
3. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In SemEval. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Claire+Cardie%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Weiwei+Guo%2C+Rada+Mihalcea%2C+German+Rigau%2C+and+Janyce+Wiebe%2E+2014%2E+Semeval%2D2014+task+10%3A+Multilingual+semantic+te) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2014-semeval-2014-task-10-multilingual/)
4. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In LREC. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Douwe+Kiela%2E+2018%2E+SentEval%3A+An+evaluation+toolkit+for+universal+sentence+representations%2E+In+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/)
5. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+learning+of+universal+sentence+representations+from+natural+language+inference+data%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2E)
7. Hongchao Fang and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766. — [arXiv:2005.12766](https://arxiv.org/abs/2005.12766) [find](https://scholar.google.com/scholar?q=Hongchao+Fang+and+Pengtao+Xie%2E+2020%2E+Cert%3A+Contrastive+self%2Dsupervised+learning+for+language+understanding%2E+arXiv+preprint+arXiv%3A2005%2E12766%2E+%E2%80%94+%5BarXiv%3A2005%2E12766%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2005%2E12766%29)
8. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Xingcheng+Yao%2C+and+Danqi+Chen%2E+2021%2E+Simcse%3A+Simple+contrastive+learning+of+sentence+embeddings%2E+arXiv+preprint+arXiv%3A2104%2E08821%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/)
9. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Rada+Mihalcea%2C+German+Rigau%2C+and+Janyce+Wiebe%2E+2016%2E+SemEval%2D2016+task+1%3A+Semantic+textual+similarity%2C+monolingual+and+cross%2Dl) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/)
10. John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. 2020. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659. [find](https://scholar.google.com/scholar?q=John+M+Giorgi%2C+Osvald+Nitski%2C+Gary+D+Bader%2C+and+Bo+Wang%2E+2020%2E+Declutr%3A+Deep+contrastive+learning+for+unsupervised+textual+representations%2E+arXiv+preprint+arXiv%3A2006%2E03659%2E) [our card](https://md.research.elamur.ai/free/en/papers/giorgi-2021-declutr-deep-contrastive-learning-unsupervised/)
11. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In SemEval. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Daniel+Cer%2C+Mona+Diab%2C+and+Aitor+Gonzalez%2DAgirre%2E+2012%2E+SemEval%2D2012+task+6%3A+A+pilot+on+semantic+textual+similarity%2E+In+SemEval%2E) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/)
12. Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In ICLR. [find](https://scholar.google.com/scholar?q=Beliz+Gunel%2C+Jingfei+Du%2C+Alexis+Conneau%2C+and+Veselin+Stoyanov%2E+2021%2E+Supervised+contrastive+learning+for+pre%2Dtrained+language+model+fine%2Dtuning%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/gunel-2020-supervised-contrastive-learning-pre-trained/)
13. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In *SEM. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+and+Weiwei+Guo%2E+2013%2E+%2ASEM+2013+shared+task%3A+Semantic+textual+similarity%2E+In+%2ASEM%2E) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2013-sem-2013-shared-task-semantic/)
14. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In CVPR. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Haoqi+Fan%2C+Yuxin+Wu%2C+Saining+Xie%2C+and+Ross+Girshick%2E+2020%2E+Momentum+contrast+for+unsupervised+visual+representation+learning%2E+In+CVPR%2E) [our card](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/)
15. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Samuel+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+EMNLP%2E)
16. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In ICML. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
17. Fredrik Carlsson, Evangelia Gogoulou, Erik Ylipää, Amaru Cuba Gyllensten, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In ICLR. [find](https://scholar.google.com/scholar?q=Fredrik+Carlsson%2C+Evangelia+Gogoulou%2C+Erik+Ylipa%CC%88a%CC%88%2C+Amaru+Cuba+Gyllensten%2C+and+Magnus+Sahlgren%2E+2021%2E+Semantic+re%2Dtuning+with+contrastive+tension%2E+In+ICLR%2E)
18. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+In%CC%83igo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+SemEval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+SemEval%2E)
19. Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. 2020. A survey on contrastive self-supervised learning. arXiv preprint arXiv:2011.00362. — [arXiv:2011.00362](https://arxiv.org/abs/2011.00362) [find](https://scholar.google.com/scholar?q=Ashish+Jaiswal%2C+Ashwin+Ramesh+Babu%2C+Mohammad+Zaki+Zadeh%2C+Debapriya+Banerjee%2C+and+Fillia+Makedon%2E+2020%2E+A+survey+on+contrastive+self%2Dsupervised+learning%2E+arXiv+preprint+arXiv%3A2011%2E00362%2E+%E2%80%94+%5BarXiv%3A2011%2E)
20. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In ACL. [find](https://scholar.google.com/scholar?q=Ganesh+Jawahar%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Djame%CC%81+Seddah%2E+2019%2E+What+does+BERT+learn+about+the+structure+of+language%3F+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/)
21. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Yinfei+Yang%2C+Sheng%2Dyi+Kong%2C+Nan+Hua%2C+Nicole+Limtiaco%2C+Rhomni+St+John%2C+Noah+Constant%2C+Mario+Guajardo%2DCespedes%2C+Steve+Yuan%2C+Chris+Tar%2C+et+al%2E+2018%2E+Universal+sentence+encoder%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/)
22. Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In ACL. [find](https://scholar.google.com/scholar?q=Haoming+Jiang%2C+Pengcheng+He%2C+Weizhu+Chen%2C+Xiaodong+Liu%2C+Jianfeng+Gao%2C+and+Tuo+Zhao%2E+2020%2E+SMART%3A+Robust+and+efficient+fine%2Dtuning+for+pre%2Dtrained+natural+language+models+through+principled+regularized) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-smart-robust-efficient-fine-tuning/)
23. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In ICML. [find](https://scholar.google.com/scholar?q=Ting+Chen%2C+Simon+Kornblith%2C+Mohammad+Norouzi%2C+and+Geoffrey+Hinton%2E+2020%2E+A+simple+framework+for+contrastive+learning+of+visual+representations%2E+In+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/)
24. Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. 2020. Contrastive representation learning: A framework and review. IEEE Access. [find](https://scholar.google.com/scholar?q=Phuc+H+Le%2DKhac%2C+Graham+Healy%2C+and+Alan+F+Smeaton%2E+2020%2E+Contrastive+representation+learning%3A+A+framework+and+review%2E+IEEE+Access%2E)
25. Xinlei Chen and Kaiming He. 2020. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566. — [arXiv:2011.10566](https://arxiv.org/abs/2011.10566) [find](https://scholar.google.com/scholar?q=Xinlei+Chen+and+Kaiming+He%2E+2020%2E+Exploring+simple+siamese+representation+learning%2E+arXiv+preprint+arXiv%3A2011%2E10566%2E+%E2%80%94+%5BarXiv%3A2011%2E10566%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2011%2E10566%29)
26. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In EMNLP. [find](https://scholar.google.com/scholar?q=Bohan+Li%2C+Hao+Zhou%2C+Junxian+He%2C+Mingxuan+Wang%2C+Yiming+Yang%2C+and+Lei+Li%2E+2020%2E+On+the+sentence+embeddings+from+pre%2Dtrained+language+models%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
28. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In ICLR. [find](https://scholar.google.com/scholar?q=Lajanugen+Logeswaran+and+Honglak+Lee%2E+2018%2E+An+efficient+framework+for+learning+sentence+representations%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/)
29. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In LREC. [find](https://scholar.google.com/scholar?q=Marco+Marelli%2C+Stefano+Menini%2C+Marco+Baroni%2C+Luisa+Bentivogli%2C+Raffaella+Bernardi%2C+and+Roberto+Zamparelli%2E+2014%2E+A+SICK+cure+for+the+evaluation+of+compositional+distributional+semantic+models%2E+In+LREC)
30. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. NeurIPS. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+NeurIPS%2E)
31. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence. [find](https://scholar.google.com/scholar?q=Takeru+Miyato%2C+Shin%2Dichi+Maeda%2C+Masanori+Koyama%2C+and+Shin+Ishii%2E+2018%2E+Virtual+adversarial+training%3A+a+regularization+method+for+supervised+and+semi%2Dsupervised+learning%2E+IEEE+transactions+on+pattern+a)
32. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884. [find](https://scholar.google.com/scholar?q=Marius+Mosbach%2C+Maksym+Andriushchenko%2C+and+Dietrich+Klakow%2E+2020%2E+On+the+stability+of+fine%2Dtuning+bert%3A+Misconceptions%2C+explanations%2C+and+strong+baselines%2E+arXiv+preprint+arXiv%3A2006%2E04884%2E)
33. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). [find](https://scholar.google.com/scholar?q=Nathan+Ng%2C+Kyra+Yee%2C+Alexei+Baevski%2C+Myle+Ott%2C+Michael+Auli%2C+and+Sergey+Edunov%2E+2019%2E+Facebook+fair%E2%80%99s+wmt19+news+translation+task+submission%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+Transla) [our card](https://md.research.elamur.ai/free/en/papers/ng-2019-facebook-fair-s-wmt19-news/)
34. Bin Wang and C-C Jay Kuo. 2020. Sbert-wk: A sentence embedding method by dissecting bert-based word models. arXiv preprint arXiv:2002.06652. — [arXiv:2002.06652](https://arxiv.org/abs/2002.06652) [find](https://scholar.google.com/scholar?q=Bin+Wang+and+C%2DC+Jay+Kuo%2E+2020%2E+Sbert%2Dwk%3A+A+sentence+embedding+method+by+dissecting+bert%2Dbased+word+models%2E+arXiv+preprint+arXiv%3A2002%2E06652%2E+%E2%80%94+%5BarXiv%3A2002%2E06652%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E06652%29)
35. Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning. arXiv preprint arXiv:2104.06979. — [arXiv:2104.06979](https://arxiv.org/abs/2104.06979) [find](https://scholar.google.com/scholar?q=Kexin+Wang%2C+Nils+Reimers%2C+and+Iryna+Gurevych%2E+2021%2E+Tsdae%3A+Using+transformer%2Dbased+sequential+denoising+auto%2Dencoder+for+unsupervised+sentence+embedding+learning%2E+arXiv+preprint+arXiv%3A2104%2E06979%2E+%E2%80%94+%5Ba)
36. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2E)
37. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+et+al%2E+2019%2E+Huggingface%E2%80%99s+transformers%3A+State%2Dof%2Dth) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
38. Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466. — [arXiv:2012.15466](https://arxiv.org/abs/2012.15466) [find](https://scholar.google.com/scholar?q=Zhuofeng+Wu%2C+Sinong+Wang%2C+Jiatao+Gu%2C+Madian+Khabsa%2C+Fei+Sun%2C+and+Hao+Ma%2E+2020%2E+Clear%3A+Contrastive+learning+for+sentence+representation%2E+arXiv+preprint+arXiv%3A2012%2E15466%2E+%E2%80%94+%5BarXiv%3A2012%2E15466%5D%28https%3A%2F%2Far)
39. Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. In NeurIPS. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Zihang+Dai%2C+Eduard+Hovy%2C+Thang+Luong%2C+and+Quoc+Le%2E+2020%2E+Unsupervised+data+augmentation+for+consistency+training%2E+In+NeurIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)
40. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In ICLR. [find](https://scholar.google.com/scholar?q=Chen+Zhu%2C+Yu+Cheng%2C+Zhe+Gan%2C+Siqi+Sun%2C+Tom+Goldstein%2C+and+Jingjing+Liu%2E+2020%2E+Freelb%3A+Enhanced+adversarial+training+for+natural+language+understanding%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2019-freelb-enhanced-adversarial-training-natural/)
41. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+EMNLP%2E)
42. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In EMNLP-IJCNLP. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+Siamese+BERT%2Dnetworks%2E+In+EMNLP%2DIJCNLP%2E)
43. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+ACL%2E)
44. Junfeng Tian, Zhiheng Zhou, Man Lan, and Yuanbin Wu. 2017. ECNU at SemEval-2017 task 1: Leverage kernel-based traditional NLP features and neural networks to build a universal model for multilingual and cross-lingual semantic textual similarity. In Proceedings of the 11th International Workshop on S… [find](https://scholar.google.com/scholar?q=Junfeng+Tian%2C+Zhiheng+Zhou%2C+Man+Lan%2C+and+Yuanbin+Wu%2E+2017%2E+ECNU+at+SemEval%2D2017+task+1%3A+Leverage+kernel%2Dbased+traditional+NLP+features+and+neural+networks+to+build+a+universal+model+for+multilingual+a)
45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NeurIPS%2E)
46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)