---
title: "DiffCSE: Difference-based Contrastive Learning — Chuang 2022"
description: "Unsupervised sentence embedding methods such as SimCSE train an encoder so that two dropout-perturbed copies of a sentence stay close together."
authors: ["Yung-Sung Chuang", "Rumen Dangovski", "Hongyin Luo", "Yang Zhang", "Shiyu Chang", "Marin Soljacic", "Shang-Wen Li", "Scott Yih", "Yoon Kim", "James Glass"]
year: 2022
venue: "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2022.naacl-main.311/"
doi: "10.18653/v1/2022.naacl-main.311"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chuang-2022-diffcse-difference-based-contrastive-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chuang-2022-diffcse-difference-based-contrastive-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

# DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

**Authors:** [Yung-Sung Chuang](https://md.research.elamur.ai/free/en/authors/yung-sung-chuang/), Rumen Dangovski, [Hongyin Luo](https://md.research.elamur.ai/free/en/authors/hongyin-luo/), [Yang Zhang](https://md.research.elamur.ai/free/en/authors/yang-zhang/), Shiyu Chang, Marin Soljacic, Shang-Wen Li, [Scott Yih](https://md.research.elamur.ai/free/en/authors/scott-yih/), [Yoon Kim](https://md.research.elamur.ai/free/en/authors/yoon-kim/), [James Glass](https://md.research.elamur.ai/free/en/authors/james-glass/) · **Year:** 2022 · **Venue:** Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.naacl-main.311/) · [PDF](https://aclanthology.org/2022.naacl-main.311.pdf) · [DOI 10.18653/v1/2022.naacl-main.311](https://doi.org/10.18653/v1/2022.naacl-main.311) · [OpenAlex](https://openalex.org/W4224313754)

**Citations:** 165 citations (openalex-snapshot · 2026-10-14) · [104](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Unsupervised sentence embedding methods such as SimCSE train an encoder so that two dropout-perturbed copies of a sentence stay close together. Surface edits like word replacement often change meaning, so forcing the encoder to ignore them hurts quality. DiffCSE takes a different approach. Instead of discarding such edits, it makes the encoder sensitive to them, framing this as equivariant contrastive learning carried over from computer vision.

The method keeps SimCSE's dropout-based contrastive loss. It adds a conditional ELECTRA-style objective: tokens of a sentence are randomly masked, a frozen DistilBERT/DistilRoBERTa generator fills them in, and a discriminator, given the sentence embedding, must flag which tokens were replaced. Because gradients from this task reach the encoder, the embedding is pushed to encode fine-grained meaning.

The encoders are initialised from BERT and RoBERTa and evaluated on 7 STS tasks and 7 SentEval transfer tasks. DiffCSE beats unsupervised SimCSE by about 2.3 points on STS, setting a new unsupervised state of the art. It also shows that augmentations once considered harmful can still be useful in NLP.

## Key points
- The training loss adds the contrastive term to λ times the replaced-token-detection term, and λ = 0.005 worked best. Setting λ to zero recovers plain SimCSE.
- The generator stays frozen during training. After training, the discriminator is thrown away and only the encoder output (a [CLS] embedding passed through an MLP with BatchNorm) is used.
- On retrieval over the 2758-sentence STS-B test set (97 fully similar pairs), DiffCSE-BERT base scores recall@1/5/10 of 78.87/95.36/97.42. The SimCSE scores are 77.84/92.78/95.88.
- In the representation-space analysis, DiffCSE has much better alignment than SimCSE (0.097 vs 0.177) but weaker uniformity (-1.438 vs -2.313). This suggests its gains come from alignment. The average STS score is 78.49 for DiffCSE versus 76.16 for SimCSE.
- Code and pretrained models are available at github.com/voidism/DiffCSE. The supervised NLI-based setting was not explored.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

104 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 103 not yet.

- [Mitigating Language Bias in Multilingual Sentence Embeddings for Cross-Lingual Similarity Estimation](https://md.research.elamur.ai/free/en/papers/nonomura-2026-mitigating-language-bias-multilingual-sentence/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chuang2022diffcse,
  title = {{DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings}},
  author = {Chuang, Yung-Sung and Dangovski, Rumen and Luo, Hongyin and Zhang, Yang and Chang, Shiyu and Soljacic, Marin and Li, Shang-Wen and Yih, Scott and Kim, Yoon and Glass, James},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.311},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.311}
}
```

- [← previous Detection of Negative Campaign in Israeli Municipal Elections](https://md.research.elamur.ai/free/en/papers/aclbulk-2022-litvak-detection-negative-campaign-israeli-municipal/)
- [next → Diffusion-LM Improves Controllable Text Generation](https://md.research.elamur.ai/free/en/papers/li-2022-diffusion-lm-improves-controllable-text/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

40 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval… [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Rada+Mihalcea%2C+German+Rigau%2C+and+Janyce+Wiebe%2E+2016%2E+SemEval%2D2016+task+1%3A+Semantic+textual+similarity%2C+monolingual+and+cross%2Dl) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/)
2. Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. [find](https://scholar.google.com/scholar?q=Fredrik+Carlsson%2C+Amaru+Cuba+Gyllensten%2C+Evangelia+Gogoulou%2C+Erik+Ylip%C3%A4%C3%A4+Hellqvist%2C+and+Magnus+Sahlgren%2E+2021%2E+Semantic+re%2Dtuning+with+contrastive+tension%2E)
3. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.… [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+I%C3%B1igo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+SemEval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+Proceedings+of+th)
4. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demo… [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Yinfei+Yang%2C+Sheng%2Dyi+Kong%2C+Nan+Hua%2C+Nicole+Limtiaco%2C+Rhomni+St+John%2C+Noah+Constant%2C+Mario+Guajardo%2DCespedes%2C+Steve+Yuan%2C+Chris+Tar%2C+et+al%2E+2018%2E+Universal+sentence+encoder+for+english%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder-english/)
5. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR. [find](https://scholar.google.com/scholar?q=Ting+Chen%2C+Simon+Kornblith%2C+Mohammad+Norouzi%2C+and+Geoffrey+Hinton%2E+2020%2E+A+simple+framework+for+contrastive+learning+of+visual+representations%2E+In+International+conference+on+machine+learning%2C+pages+1) [our card](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/)
6. Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758. [find](https://scholar.google.com/scholar?q=Xinlei+Chen+and+Kaiming+He%2E+2021%2E+Exploring+simple+siamese+representation+learning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages+15750%E2%80%9315758%2E)
7. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+arXiv+preprint+arXiv%3A2003%2E10555%2E)
8. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Douwe+Kiela%2E+2018%2E+SentEval%3A+An+evaluation+toolkit+for+universal+sentence+representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/)
9. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C3%AFc+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+learning+of+universal+sentence+representations+from+natural+language+inference+data%2E+In+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
10. Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačić. 2021. Equivariant contrastive learning. arXiv preprint arXiv:2111.00899. — [arXiv:2111.00899](https://arxiv.org/abs/2111.00899) [find](https://scholar.google.com/scholar?q=Rumen+Dangovski%2C+Li+Jing%2C+Charlotte+Loh%2C+Seungwook+Han%2C+Akash+Srivastava%2C+Brian+Cheung%2C+Pulkit+Agrawal%2C+and+Marin+Soljac%CC%8Cic%CC%81%2E+2021%2E+Equivariant+contrastive+learning%2E+arXiv+preprint+arXiv%3A2111%2E00899%2E+%E2%80%94)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
12. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=William+B+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
13. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Xingcheng+Yao%2C+and+Danqi+Chen%2E+2021%2E+Simcse%3A+Simple+contrastive+learning+of+sentence+embeddings%2E+arXiv+preprint+arXiv%3A2104%2E08821%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/)
14. John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. 2020. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659. [find](https://scholar.google.com/scholar?q=John+M+Giorgi%2C+Osvald+Nitski%2C+Gary+D+Bader%2C+and+Bo+Wang%2E+2020%2E+Declutr%3A+Deep+contrastive+learning+for+unsupervised+textual+representations%2E+arXiv+preprint+arXiv%3A2006%2E03659%2E) [our card](https://md.research.elamur.ai/free/en/papers/giorgi-2021-declutr-deep-contrastive-learning-unsupervised/)
15. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. 2020. Bootstrap your own latent - a new approach to self-su… [find](https://scholar.google.com/scholar?q=Jean%2DBastien+Grill%2C+Florian+Strub%2C+Florent+Altch%C3%A9%2C+Corentin+Tallec%2C+Pierre+Richemond%2C+Elena+Buchatskaya%2C+Carl+Doersch%2C+Bernardo+Avila+Pires%2C+Zhaohan+Guo%2C+Mohammad+Gheshlaghi+Azar%2C+Bilal+Piot%2C+koray+ka)
16. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Kyunghyun+Cho%2C+and+Anna+Korhonen%2E+2016%2E+Learning+distributed+representations+of+sentences+from+unlabelled+data%2E+arXiv+preprint+arXiv%3A1602%2E03483%2E) [our card](https://md.research.elamur.ai/free/en/papers/hill-2016-learning-distributed-representations-sentences-unlabelled/)
17. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+Proceedings+of+the+tenth+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+pages+168%E2%80%93177%2E)
18. Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. 2021. On feature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9598–9608. [find](https://scholar.google.com/scholar?q=Tianyu+Hua%2C+Wenxiao+Wang%2C+Zihui+Xue%2C+Sucheng+Ren%2C+Yue+Wang%2C+and+Hang+Zhao%2E+2021%2E+On+feature+decorrelation+in+self%2Dsupervised+learning%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Comput)
19. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. PMLR. [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+2015%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+International+conference+on+machine+learning%2C+pages+448%E2%80%93456%2E+)
20. Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for bert sentence representations. arXiv preprint arXiv:2106.07345. [find](https://scholar.google.com/scholar?q=Taeuk+Kim%2C+Kang+Min+Yoo%2C+and+Sang%2Dgoo+Lee%2E+2021%2E+Self%2Dguided+contrastive+learning+for+bert+sentence+representations%2E+arXiv+preprint+arXiv%3A2106%2E07345%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/)
21. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. pages 3294–3302. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+Salakhutdinov%2C+Richard+S+Zemel%2C+Antonio+Torralba%2C+Raquel+Urtasun%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+pages+3294%E2%80%933302%2E)
22. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119–9130, Online. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Bohan+Li%2C+Hao+Zhou%2C+Junxian+He%2C+Mingxuan+Wang%2C+Yiming+Yang%2C+and+Lei+Li%2E+2020%2E+On+the+sentence+embeddings+from+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/)
23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
24. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. [find](https://scholar.google.com/scholar?q=Lajanugen+Logeswaran+and+Honglak+Lee%2E+2018%2E+An+efficient+framework+for+learning+sentence+representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/)
25. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compositional distributional semantic models. In Lrec, pages 216–223. Reykjavik. [find](https://scholar.google.com/scholar?q=Marco+Marelli%2C+Stefano+Menini%2C+Marco+Baroni%2C+Luisa+Bentivogli%2C+Raffaella+Bernardi%2C+Roberto+Zamparelli%2C+et+al%2E+2014%2E+A+sick+cure+for+the+evaluation+of+compositional+distributional+semantic+models%2E+In+L)
26. Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021. Coco-lm: Correcting and contrasting text sequences for language model pretraining. arXiv preprint arXiv:2102.08473. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Chenyan+Xiong%2C+Payal+Bajaj%2C+Saurabh+Tiwary%2C+Paul+Bennett%2C+Jiawei+Han%2C+and+Xia+Song%2E+2021%2E+Coco%2Dlm%3A+Correcting+and+contrasting+text+sequences+for+language+model+pretraining%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/meng-2021-coco-lm-correcting-contrasting-text/)
27. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cs/0409058. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2004%2E+A+sentimental+education%3A+Sentiment+analysis+using+subjectivity+summarization+based+on+minimum+cuts%2E+arXiv+preprint+cs%2F0409058%2E) [our card](https://md.research.elamur.ai/free/en/papers/pang-2004-sentimental-education-sentiment-analysis-subjectivity/)
28. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 115–124. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2005%2E+Seeing+stars%3A+Exploiting+class+relationships+for+sentiment+categorization+with+respect+to+rating+scales%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for) [our card](https://md.research.elamur.ai/free/en/papers/pang-2005-seeing-stars-exploiting-class-relationships/)
29. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+proce)
30. Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316. — [arXiv:2103.15316](https://arxiv.org/abs/2103.15316) [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Jiarun+Cao%2C+Weijie+Liu%2C+and+Yangyiwen+Ou%2E+2021%2E+Whitening+sentence+representations+for+better+semantics+and+faster+retrieval%2E+arXiv+preprint+arXiv%3A2103%2E15316%2E+%E2%80%94+%5BarXiv%3A2103%2E15316%5D%28https%3A%2F%2F)
31. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962. [find](https://scholar.google.com/scholar?q=Iulia+Turc%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Well%2Dread+students+learn+better%3A+On+the+importance+of+pre%2Dtraining+compact+models%2E+arXiv+preprint+arXiv%3A1908%2E08962%2E) [our card](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/)
32. Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200–207. [find](https://scholar.google.com/scholar?q=Ellen+M+Voorhees+and+Dawn+M+Tice%2E+2000%2E+Building+a+question+answering+test+collection%2E+In+Proceedings+of+the+23rd+annual+international+ACM+SIGIR+conference+on+Research+and+development+in+information+r)
33. Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929–9939. PMLR. [find](https://scholar.google.com/scholar?q=Tongzhou+Wang+and+Phillip+Isola%2E+2020%2E+Understanding+contrastive+representation+learning+through+alignment+and+uniformity+on+the+hypersphere%2E+In+International+Conference+on+Machine+Learning%2C+pages+992)
34. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165–210. [find](https://scholar.google.com/scholar?q=Janyce+Wiebe%2C+Theresa+Wilson%2C+and+Claire+Cardie%2E+2005%2E+Annotating+expressions+of+opinions+and+emotions+in+language%2E+Language+resources+and+evaluation%2C+39%282%29%3A165%E2%80%93210%2E)
35. Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A contrastive framework for self-supervised sentence representation transfer. arXiv preprint arXiv:2105.11741. [find](https://scholar.google.com/scholar?q=Yuanmeng+Yan%2C+Rumei+Li%2C+Sirui+Wang%2C+Fuzheng+Zhang%2C+Wei+Wu%2C+and+Weiran+Xu%2E+2021%2E+Consert%3A+A+contrastive+framework+for+self%2Dsupervised+sentence+representation+transfer%2E+arXiv+preprint+arXiv%3A2105%2E11741%2E) [our card](https://md.research.elamur.ai/free/en/papers/yan-2021-consert-contrastive-framework-self-supervised/)
36. Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, and Eric Darve. 2020. Universal sentence representation learning with conditional masked language model. arXiv preprint arXiv:2012.14388. [find](https://scholar.google.com/scholar?q=Ziyi+Yang%2C+Yinfei+Yang%2C+Daniel+Cer%2C+Jax+Law%2C+and+Eric+Darve%2E+2020%2E+Universal+sentence+representation+learning+with+conditional+masked+language+model%2E+arXiv+preprint+arXiv%3A2012%2E14388%2E)
37. Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1601–1610. [find](https://scholar.google.com/scholar?q=Yan+Zhang%2C+Ruidan+He%2C+Zuozhu+Liu%2C+Kwan+Hui+Lim%2C+and+Lidong+Bing%2E+2020%2E+An+unsupervised+sentence+embedding+method+by+mutual+information+maximization%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2020-unsupervised-sentence-embedding-method-mutual/)
38. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the)
39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2E)
40. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Y+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+I)