---
title: "On the Sentence Embeddings from Pre-trained — Li 2020"
description: "When BERT is used without fine-tuning, its sentence vectors (averaged context embeddings from the last layers) are poor at semantic textual similarity…"
authors: ["Bohan Li", "Hao Zhou", "Junxian He", "Mingxuan Wang", "Yiming Yang", "Lei Li"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.733/"
doi: "10.18653/v1/2020.emnlp-main.733"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / On the Sentence Embeddings from Pre-trained Language Models

# On the Sentence Embeddings from Pre-trained Language Models

**Authors:** Bohan Li, [Hao Zhou](https://md.research.elamur.ai/free/en/authors/hao-zhou/), [Junxian He](https://md.research.elamur.ai/free/en/authors/junxian-he/), [Mingxuan Wang](https://md.research.elamur.ai/free/en/authors/mingxuan-wang/), [Yiming Yang](https://md.research.elamur.ai/free/en/authors/yiming-yang/), [Lei Li](https://md.research.elamur.ai/free/en/authors/lei-li/) · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.733/) · [PDF](https://aclanthology.org/2020.emnlp-main.733.pdf) · [DOI 10.18653/v1/2020.emnlp-main.733](https://doi.org/10.18653/v1/2020.emnlp-main.733) · [OpenAlex](https://openalex.org/W3105816068)

**Citations:** 572 citations (openalex-snapshot · 2026-10-14) · [258](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
When BERT is used without fine-tuning, its sentence vectors (averaged context embeddings from the last layers) are poor at semantic textual similarity and can score below averaged GloVe. The authors ask whether these vectors lack semantic content or whether that content is simply hard to reach. They first argue theoretically that the masked language model objective, through PMI-like co-occurrence statistics, should yield semantically meaningful context embeddings. Empirical probing then shows the embedding space is anisotropic and shaped by word frequency: frequent words cluster near the origin, while rare ones lie far away in sparse regions. To fix this, they propose BERT-flow. A normalizing flow is trained without labels to map frozen BERT sentence embeddings onto a standard isotropic Gaussian, and cosine similarity is then computed in that latent space. On seven STS benchmarks this unsupervised calibration consistently improves BERT. Combined with NLI supervision, it surpasses Sentence-BERT. The work shows that pre-trained encoders hold more usable semantics than naive pooling reveals, which matters wherever labeled similarity data is scarce.

## Key points
- Without any downstream supervision, the flow transformation improves BERT on 7 STS tasks by an average of 8.16 Spearman points, with a maximum gain of 12.70.
- Only the flow network is trained (maximum likelihood, unsupervised), and BERT weights stay frozen. Code is available at github.com/bohanli/BERT-flow.
- On STS-B, BERT-base-flow (target) scores 65.62. The best tuned combination of the isotropy baselines, standard normalization plus NATSV, reaches 63.51.
- Frequency bias shows up directly in the norms: the mean ℓ2-norm of BERT word embeddings grows from 0.95 for the most frequent words to 1.45 for rarer ones.
- BERT similarity tracks lexical overlap too closely. Its correlation with edit distance on STS-B is ρ = −50.49, against −24.61 for gold labels and −28.01 after the flow. The method was also tested on QNLI without task-specific training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

258 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 247 not yet.

- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/chuang-2022-diffcse-difference-based-contrastive-learning/)
- [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://md.research.elamur.ai/free/en/papers/dai-2023-auggpt-leveraging-chatgpt-text-data/)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/)
- [MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection](https://md.research.elamur.ai/free/en/papers/hbo-2026-smiley-miqrabert-regression-based-sentence-bert/)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://md.research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://md.research.elamur.ai/free/en/papers/meng-2021-coco-lm-correcting-contrasting-text/)
- [Text and Code Embeddings by Contrastive Pre-Training](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://md.research.elamur.ai/free/en/papers/park-2023-linear-representation-hypothesis-geometry-large/)
- [Generating Datasets with Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://md.research.elamur.ai/free/en/papers/yan-2021-consert-contrastive-framework-self-supervised/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2020sentence,
  title = {{On the Sentence Embeddings from Pre-trained Language Models}},
  author = {Li, Bohan and Zhou, Hao and He, Junxian and Wang, Mingxuan and Yang, Yiming and Li, Lei},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.733},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.733}
}
```

- [← previous On the Importance of Tokenization in Arabic Embedding Models](https://md.research.elamur.ai/free/en/papers/aclbulk-2020-alkaoud-importance-tokenization-arabic-embedding-models/)
- [next → On the syntax of object pronouns in Old English and Early Middle English](https://md.research.elamur.ai/free/en/papers/bastiani-2020-syntax-object-pronouns-old-english/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

35 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015. SemEval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedings of SemEval. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Claire+Cardie%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Weiwei+Guo%2C+Inigo+Lopez%2DGazpio%2C+Montse+Maritxalar%2C+Rada+Mihalcea%2C+et+al%2E+2015%2E+SemEval%2D2015+task+2%3A+Semantic+te) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/)
2. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of SemEval. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Claire+Cardie%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+Weiwei+Guo%2C+Rada+Mihalcea%2C+German+Rigau%2C+and+Janyce+Wiebe%2E+2014%2E+SemEval%2D2014+task+10%3A+Multilingual+semantic+te) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2014-semeval-2014-task-10-multilingual/)
3. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of SemEval. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Carmen+Banea%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez+Agirre%2C+Rada+Mihalcea%2C+German+Rigau+Claramunt%2C+and+Janyce+Wiebe%2E+2016%2E+SemEval%2D2016+task+1%3A+Semantic+textual+similarity%2C+monolingual+a) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/)
4. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of SemEval. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Daniel+Cer%2C+Mona+Diab%2C+and+Aitor+Gonzalez%2DAgirre%2E+2012%2E+SemEval%2D2012+task+6%3A+A+pilot+on+semantic+textual+similarity%2E+In+Proceedings+of+SemEval%2E) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/)
5. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. * SEM 2013 shared task: Semantic textual similarity. In Proceedings of SemEval. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Daniel+Cer%2C+Mona+Diab%2C+Aitor+Gonzalez%2DAgirre%2C+and+Weiwei+Guo%2E+2013%2E+%2A+SEM+2013+shared+task%3A+Semantic+textual+similarity%2E+In+Proceedings+of+SemEval%2E) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2013-sem-2013-shared-task-semantic/)
6. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Sanjeev+Arora%2C+Yingyu+Liang%2C+and+Tengyu+Ma%2E+2017%2E+A+simple+but+tough%2Dto%2Dbeat+baseline+for+sentence+embeddings%2E+In+Proceedings+of+ICLR%2E)
7. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+EMNLP%2E)
8. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+Inigo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+SemEval%2D2017+task+1%3A+Semantic+textual+similarity%2Dmultilingual+and+cross%2Dlingual+focused+evaluation%2E+arXiv+preprint+arXi)
9. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Yinfei+Yang%2C+Sheng%2Dyi+Kong%2C+Nan+Hua%2C+Nicole+Limtiaco%2C+Rhomni+St+John%2C+Noah+Constant%2C+Mario+Guajardo%2DCespedes%2C+Steve+Yuan%2C+Chris+Tar%2C+et+al%2E+2018%2E+Universal+sentence+encoder%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/)
10. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Douwe+Kiela%2E+2018%2E+SentEval%3A+An+evaluation+toolkit+for+universal+sentence+representations%2E+In+Proceedings+of+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/)
11. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C3%AFc+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+learning+of+universal+sentence+representations+from+natural+language+inference+data%2E+In+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2E)
13. Laurent Dinh, David Krueger, and Yoshua Bengio. 2015. NICE: Non-linear independent components estimation. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Laurent+Dinh%2C+David+Krueger%2C+and+Yoshua+Bengio%2E+2015%2E+NICE%3A+Non%2Dlinear+independent+components+estimation%2E+In+Proceedings+of+ICLR%2E)
14. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of EMNLP-IJCNLP. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+comparing+the+geometry+of+bert%2C+elmo%2C+and+gpt%2D2+embeddings%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2E)
15. Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Towards understanding linear word analogies. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+David+Duvenaud%2C+and+Graeme+Hirst%2E+2019%2E+Towards+understanding+linear+word+analogies%2E+In+Proceedings+of+ACL%2E)
16. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Representation degeneration problem in training natural language generation models. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Jun+Gao%2C+Di+He%2C+Xu+Tan%2C+Tao+Qin%2C+Liwei+Wang%2C+and+Tie%2DYan+Liu%2E+2019%2E+Representation+degeneration+problem+in+training+natural+language+generation+models%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/)
17. Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. 2020. From variational to deterministic autoencoders. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Partha+Ghosh%2C+Mehdi+SM+Sajjadi%2C+Antonio+Vergari%2C+Michael+Black%2C+and+Bernhard+Scholkopf%2E+2020%2E+From+variational+to+deterministic+autoencoders%2E+In+Proceedings+of+ICLR%2E)
18. Durk P Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible 1x1 convolutions. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Durk+P+Kingma+and+Prafulla+Dhariwal%2E+2018%2E+Glow%3A+Generative+flow+with+invertible+1x1+convolutions%2E+In+Proceedings+of+NeurIPS%2E)
19. Ivan Kobyzev, Simon Prince, and Marcus A Brubaker. 2019. Normalizing flows: Introduction and ideas. arXiv preprint arXiv:1908.09257. — [arXiv:1908.09257](https://arxiv.org/abs/1908.09257) [find](https://scholar.google.com/scholar?q=Ivan+Kobyzev%2C+Simon+Prince%2C+and+Marcus+A+Brubaker%2E+2019%2E+Normalizing+flows%3A+Introduction+and+ideas%2E+arXiv+preprint+arXiv%3A1908%2E09257%2E+%E2%80%94+%5BarXiv%3A1908%2E09257%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1908%2E09257%29)
20. Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014%2E+Neural+word+embedding+as+implicit+matrix+factorization%2E+In+Proceedings+of+NeurIPS%2E)
21. Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2019. A surprisingly effective fix for deep latent variable modeling of text. In Proceedings of EMNLP-IJCNLP. [find](https://scholar.google.com/scholar?q=Bohan+Li%2C+Junxian+He%2C+Graham+Neubig%2C+Taylor+Berg%2DKirkpatrick%2C+and+Yiming+Yang%2E+2019%2E+A+surprisingly+effective+fix+for+deep+latent+variable+modeling+of+text%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2E)
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
23. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Marco+Marelli%2C+Stefano+Menini%2C+Marco+Baroni%2C+Luisa+Bentivogli%2C+Raffaella+Bernardi%2C+Roberto+Zamparelli%2C+et+al%2E+2014%2E+A+SICK+cure+for+the+evaluation+of+compositional+distributional+semantic+models%2E+In+P)
24. Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2017. All-but-the-top: Simple and effective postprocessing for word representations. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Jiaqi+Mu%2C+Suma+Bhat%2C+and+Pramod+Viswanath%2E+2017%2E+All%2Dbut%2Dthe%2Dtop%3A+Simple+and+effective+postprocessing+for+word+representations%2E+In+Proceedings+of+ICLR%2E)
25. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D+Manning%2E+2014%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+EMNLP%2E)
26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E)
27. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
28. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of EMNLP-IJCNLP. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+siamese+BERT%2Dnetworks%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2E)
29. Danilo Jimenez Rezende and Fabio Viola. 2018. Taming VAEs. arXiv preprint arXiv:1810.00597. — [arXiv:1810.00597](https://arxiv.org/abs/1810.00597) [find](https://scholar.google.com/scholar?q=Danilo+Jimenez+Rezende+and+Fabio+Viola%2E+2018%2E+Taming+VAEs%2E+arXiv+preprint+arXiv%3A1810%2E00597%2E+%E2%80%94+%5BarXiv%3A1810%2E00597%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1810%2E00597%29)
30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proceedings+of+NeurIPS%2E)
31. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2019%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+I) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
32. Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. 2020. Improving neural language generation with spectrum control. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Lingxiao+Wang%2C+Jing+Huang%2C+Kevin+Huang%2C+Ziniu+Hu%2C+Guangtao+Wang%2C+and+Quanquan+Gu%2E+2020%2E+Improving+neural+language+generation+with+spectrum+control%2E+In+Proceedings+of+ICLR%2E)
33. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+ACL%2E)
34. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. 2018. Breaking the softmax bottleneck: A high-rank rnn language model. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Ruslan+Salakhutdinov%2C+and+William+W+Cohen%2E+2018%2E+Breaking+the+softmax+bottleneck%3A+A+high%2Drank+rnn+language+model%2E+In+Proceedings+of+ICLR%2E)
35. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+XLNet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+Proceedings+of+NeurIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)