--- title: "Sentence-T5: Scalable Sentence Encoders from — Ni 2021" description: "The paper asks how to obtain fixed-size sentence embeddings from T5." authors: ["Jianmo Ni", "Gustavo Hernandez Abrego", "Noah Constant", "Ji Ma", "Keith B. Hall", "Daniel Cer", "Yinfei Yang"] year: 2021 venue: "arXiv:2108.08877" source: "https://arxiv.org/abs/2108.08877" doi: "10.48550/arxiv.2108.08877" arxiv: "2108.08877" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models # Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models **Authors:** [Jianmo Ni](https://md.research.elamur.ai/free/en/authors/jianmo-ni/), [Gustavo Hernandez Abrego](https://md.research.elamur.ai/free/en/authors/gustavo-hernandez-abrego/), [Noah Constant](https://md.research.elamur.ai/free/en/authors/noah-constant/), [Ji Ma](https://md.research.elamur.ai/free/en/authors/ji-ma/), [Keith B. Hall](https://md.research.elamur.ai/free/en/authors/keith-b-hall/), [Daniel Cer](https://md.research.elamur.ai/free/en/authors/daniel-cer/), [Yinfei Yang](https://md.research.elamur.ai/free/en/authors/yinfei-yang/) · **Year:** 2021 · **Venue:** arXiv:2108.08877 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2108.08877) · [PDF](https://arxiv.org/pdf/2108.08877) · [DOI 10.48550/arxiv.2108.08877](https://doi.org/10.48550/arxiv.2108.08877) · [arXiv 2108.08877](https://arxiv.org/abs/2108.08877) · [OpenAlex](https://openalex.org/W3194782062) **Citations:** 353 citations (openalex-snapshot · 2026-10-14) · [229](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper asks how to obtain fixed-size sentence embeddings from T5. T5 is an encoder-decoder model pre-trained with span corruption, and it is strong on GLUE-style tasks. However, using it directly for retrieval or clustering is costly, because every query-candidate pair would need full cross-attention inference. The authors test three pooling strategies: - the first encoder token, - the mean over encoder outputs, - the first decoder output. Each strategy is placed inside a shared-weight dual encoder and fine-tuned with an in-batch contrastive loss. Training runs in two stages: about 2 billion mined Community QA question-answer pairs first, then a 275K-example contrastive NLI set. Model sizes range from Base up to 11B parameters. The authors also build SentGLUE, which applies SentEval-style embedding evaluation to nine GLUE tasks. Larger models improve steadily on both transfer and STS. Encoder-only variants are better for transfer, while the encoder-decoder variant performs best on STS. This shows that growth in pre-trained text-to-text models can carry over to efficient embedding models. The models are released on TF Hub. ## Key points - The three variants are ST5-Enc first, ST5-Enc mean and ST5-EncDec first. Each adds a projection layer and L2 normalization, so a dot product gives cosine similarity. Training uses Adafactor with a learning rate of 0.001, τ of 0.01, and batch sizes of 512 (NLI) and 2048 (Community QA). - At 11B parameters, ST5 averages about 84.9 on SentEval STS, ahead of SimCSE-RoBERTa large (83.76) and SBERT large (76.55). Its average on transfer tasks is above 90. - SentGLUE covers CoLA, SST-2, MRPC, STS-B, QQP, MNLI-m, MNLI-mm, QNLI and RTE; WNLI was dropped as too hard. ST5-Enc 11B scores 80.07 overall, compared with 83.40 for cross-attention T5 Base. - Larger models lower the uniformity loss and raise the alignment loss only slightly. Pre-training on Community QA before NLI adds gains on top of model scaling. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 229 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 217 not yet. - [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/) - [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/) - [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/) - [Generative Representational Instruction Tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/) - [Large Dual Encoders Are Generalizable Retrievers](https://md.research.elamur.ai/free/en/papers/ni-2021-large-dual-encoders-generalizable-retrievers/) - [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://md.research.elamur.ai/free/en/papers/nussbaum-2024-nomic-embed-training-reproducible-long/) - [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://md.research.elamur.ai/free/en/papers/su-2022-one-embedder-any-task-instruction/) - [Transformer Memory as a Differentiable Search Index](https://md.research.elamur.ai/free/en/papers/tay-2022-transformer-memory-differentiable-search-index/) - [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/) - [Improving Text Embeddings with Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/) - [Multilingual E5 Text Embeddings: A Technical Report](https://md.research.elamur.ai/free/en/papers/wang-2024-multilingual-e5-text-embeddings-technical/) - [C-Pack: Packed Resources For General Chinese Embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{ni2021sentence, title = {{Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models}}, author = {Ni, Jianmo and Abrego, Gustavo Hernandez and Constant, Noah and Ma, Ji and Hall, Keith B. and Cer, Daniel and Yang, Yinfei}, year = {2021}, doi = {10.48550/arxiv.2108.08877}, eprint = {2108.08877}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2108.08877} } ``` - [← previous SemLink 2.0: Chasing Lexical Resources](https://md.research.elamur.ai/free/en/papers/stowe-2021-semlink-2-0-chasing-lexical/) - [next → Sentiment Analysis in Poems in Misurata Sub-dialect -- A Sentiment Detection in an Arabic Sub-dialect](https://md.research.elamur.ai/free/en/papers/abugharsa-2021-sentiment-analysis-poems-misurata-sub/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) ## References 34 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretabi… [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Claire+Cardie%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Weiwei+Guo%2C+I%C3%B1igo+Lopez%2DGazpio%2C+Montse+Maritxalar%2C+Rada+Mihalcea%2C+German+Rigau%2C+Larraitz+Uria%2C+and+Janyce+Wiebe) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/) 2. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In SemEval 2014. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Claire+Cardie%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Weiwei+Guo%2C+Rada+Mihalcea%2C+German+Rigau%2C+and+Janyce+Wiebe%2E+2014%2E+SemEval%2D2014+task+10%3A+Multilingual+semantic+te) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2014-semeval-2014-task-10-multilingual/) 3. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Rada+Mihalcea%2C+German+Rigau%2C+and+Janyce+Wiebe%2E+2016%2E+SemEval%2D2016+task+1%3A+Semantic+textual+similarity%2C+monolingual+and+cross%2Dl) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/) 4. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012/SemEval 2012. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Daniel+Cer%2C+Mona+Diab%2C+and+Aitor+Gonzalez%2DAgirre%2E+2012%2E+SemEval%2D2012+task+6%3A+A+pilot+on+semantic+textual+similarity%2E+In+%2ASEM+2012%2FSemEval+2012%2E) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/) 5. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In *SEM. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+and+Weiwei+Guo%2E+2013%2E+%2ASEM+2013+shared+task%3A+Semantic+textual+similarity%2E+In+%2ASEM%2E) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2013-sem-2013-shared-task-semantic/) 6. Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019. ReQA: An evaluation for end-to-end answer retrieval models. In Workshop on Machine Reading for Question Answering. [find](https://scholar.google.com/scholar?q=Amin+Ahmad%2C+Noah+Constant%2C+Yinfei+Yang%2C+and+Daniel+Cer%2E+2019%2E+ReQA%3A+An+evaluation+for+end%2Dto%2Dend+answer+retrieval+models%2E+In+Workshop+on+Machine+Reading+for+Question+Answering%2E) 7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+EMNLP%2E) 8. Daniel Matthew Cer, Mona T. Diab, Eneko Agirre, I. Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval-2016. [find](https://scholar.google.com/scholar?q=Daniel+Matthew+Cer%2C+Mona+T%2E+Diab%2C+Eneko+Agirre%2C+I%2E+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+Semeval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+SemEval%2D2) 9. Daniel Matthew Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, C. Tar, Yun-Hsuan Sung, B. Strope, and R. Kurzweil. 2018. Universal sentence encoder. In EMNLP. [find](https://scholar.google.com/scholar?q=Daniel+Matthew+Cer%2C+Yinfei+Yang%2C+Sheng+yi+Kong%2C+Nan+Hua%2C+Nicole+Limtiaco%2C+Rhomni+St%2E+John%2C+Noah+Constant%2C+Mario+Guajardo%2DCespedes%2C+Steve+Yuan%2C+C%2E+Tar%2C+Yun%2DHsuan+Sung%2C+B%2E+Strope%2C+and+R%2E+Kurzweil%2E+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/) 10. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In ICLR. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Thibault+Fevry%2C+Henry+Tsai%2C+Melvin+Johnson%2C+and+Sebastian+Ruder%2E+2021%2E+Rethinking+embedding+coupling+in+pre%2Dtrained+language+models%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/) 11. Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In LREC. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Douwe+Kiela%2E+2018%2E+Senteval%3A+An+evaluation+toolkit+for+universal+sentence+representations%2E+In+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/) 12. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C3%AFc+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+learning+of+universal+sentence+representations+from+natural+language+inference+data%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/) 13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2E) 14. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In EMNLP. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+comparing+the+geometry+of+bert%2C+elmo%2C+and+gpt%2D2+embeddings%2E+In+EMNLP%2E) 15. W. Fedus, Barret Zoph, and Noam M. Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. ArXiv, abs/2101.03961. [find](https://scholar.google.com/scholar?q=W%2E+Fedus%2C+Barret+Zoph%2C+and+Noam+M%2E+Shazeer%2E+2021%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+ArXiv%2C+abs%2F2101%2E03961%2E) 16. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. CoRR, abs/2007.01852. [find](https://scholar.google.com/scholar?q=Fangxiaoyu+Feng%2C+Yinfei+Yang%2C+Daniel+Cer%2C+Naveen+Arivazhagan%2C+and+Wei+Wang%2E+2020%2E+Language%2Dagnostic+BERT+sentence+embedding%2E+CoRR%2C+abs%2F2007%2E01852%2E) [our card](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/) 17. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. ArXiv, abs/2104.08821. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Xingcheng+Yao%2C+and+Danqi+Chen%2E+2021%2E+Simcse%3A+Simple+contrastive+learning+of+sentence+embeddings%2E+ArXiv%2C+abs%2F2104%2E08821%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/) 18. D. Gillick, A. Presta, and Gaurav Singh Tomar. 2018. End-to-end retrieval in continuous space. ArXiv, abs/1811.08008. — [arXiv:1811.08008](https://arxiv.org/abs/1811.08008) [find](https://scholar.google.com/scholar?q=D%2E+Gillick%2C+A%2E+Presta%2C+and+Gaurav+Singh+Tomar%2E+2018%2E+End%2Dto%2Dend+retrieval+in+continuous+space%2E+ArXiv%2C+abs%2F1811%2E08008%2E+%E2%80%94+%5BarXiv%3A1811%2E08008%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1811%2E08008%29) 19. Matthew Henderson, Rami Al-Rfou, B. Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and R. Kurzweil. 2017. Efficient natural language response suggestion for smart reply. ArXiv, abs/1705.00652. — [arXiv:1705.00652](https://arxiv.org/abs/1705.00652) [find](https://scholar.google.com/scholar?q=Matthew+Henderson%2C+Rami+Al%2DRfou%2C+B%2E+Strope%2C+Yun%2DHsuan+Sung%2C+L%C3%A1szl%C3%B3+Luk%C3%A1cs%2C+Ruiqi+Guo%2C+Sanjiv+Kumar%2C+Balint+Miklos%2C+and+R%2E+Kurzweil%2E+2017%2E+Efficient+natural+language+response+suggestion+for+smart+reply) 20. Ryan Kiros, Yukun Zhu, R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler. 2015. Skip-thought vectors. In NIPS. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+R%2E+Salakhutdinov%2C+R%2E+Zemel%2C+R%2E+Urtasun%2C+A%2E+Torralba%2C+and+S%2E+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+NIPS%2E) 21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+M%2E+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+ArX) 22. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In ICLR. [find](https://scholar.google.com/scholar?q=Lajanugen+Logeswaran+and+Honglak+Lee%2E+2018%2E+An+efficient+framework+for+learning+sentence+representations%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/) 23. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21/140. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+M%2E+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+W%2E+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dt) 24. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP/IJCNLP. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+In+EMNLP%2FIJCNLP%2E) 25. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In EMNLP. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2020%2E+Making+monolingual+sentence+embeddings+multilingual+using+knowledge+distillation%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/) 26. Noam M. Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML. [find](https://scholar.google.com/scholar?q=Noam+M%2E+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+ICML%2E) 27. Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neurips. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+M%2E+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Neurips%2E) 28. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understan) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) 29. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@EMNLP. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+BlackboxNLP%40EMN) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 30. Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML. [find](https://scholar.google.com/scholar?q=Tongzhou+Wang+and+Phillip+Isola%2E+2020%2E+Understanding+contrastive+representation+learning+through+alignment+and+uniformity+on+the+hypersphere%2E+In+ICML%2E) 31. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. CoRR, abs/1511.08198. [find](https://scholar.google.com/scholar?q=John+Wieting%2C+Mohit+Bansal%2C+Kevin+Gimpel%2C+and+Karen+Livescu%2E+2016%2E+Towards+universal+paraphrastic+sentence+embeddings%2E+CoRR%2C+abs%2F1511%2E08198%2E) [our card](https://md.research.elamur.ai/free/en/papers/wieting-2015-towards-universal-paraphrastic-sentence-embeddings/) 32. Yinfei Yang, Daniel Matthew Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, G. Ábrego, Steve Yuan, C. Tar, Yun-Hsuan Sung, B. Strope, and R. Kurzweil. 2020a. Multilingual universal sentence encoder for semantic retrieval. In ACL. [find](https://scholar.google.com/scholar?q=Yinfei+Yang%2C+Daniel+Matthew+Cer%2C+Amin+Ahmad%2C+Mandy+Guo%2C+Jax+Law%2C+Noah+Constant%2C+G%2E+%C3%81brego%2C+Steve+Yuan%2C+C%2E+Tar%2C+Yun%2DHsuan+Sung%2C+B%2E+Strope%2C+and+R%2E+Kurzweil%2E+2020a%2E+Multilingual+universal+sentence+encode) [our card](https://md.research.elamur.ai/free/en/papers/yang-2020-multilingual-universal-sentence-encoder-semantic/) 33. Yinfei Yang, Steve Yuan, Daniel Matthew Cer, Sheng yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, B. Strope, and R. Kurzweil. 2018. Learning semantic textual similarity from conversations. In Rep4NLP@ACL. [find](https://scholar.google.com/scholar?q=Yinfei+Yang%2C+Steve+Yuan%2C+Daniel+Matthew+Cer%2C+Sheng+yi+Kong%2C+Noah+Constant%2C+Petr+Pilar%2C+Heming+Ge%2C+Yun%2DHsuan+Sung%2C+B%2E+Strope%2C+and+R%2E+Kurzweil%2E+2018%2E+Learning+semantic+textual+similarity+from+conversati) 34. Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, and Eric Darve. 2020b. Universal sentence representation learning with conditional masked language model. [find](https://scholar.google.com/scholar?q=Ziyi+Yang%2C+Yinfei+Yang%2C+Daniel+Cer%2C+Jax+Law%2C+and+Eric+Darve%2E+2020b%2E+Universal+sentence+representation+learning+with+conditional+masked+language+model%2E)