--- title: "Towards Universal Paraphrastic Sentence — Wieting 2015" description: "Word embeddings transfer easily across tasks, but comparably reusable sentence vectors were still lacking." authors: ["John Wieting", "Mohit Bansal", "Kevin Gimpel", "Karen Livescu"] year: 2015 venue: "arXiv:1511.08198" source: "https://arxiv.org/abs/1511.08198" doi: "10.48550/arxiv.1511.08198" arxiv: "1511.08198" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wieting-2015-towards-universal-paraphrastic-sentence-embeddings/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wieting-2015-towards-universal-paraphrastic-sentence-embeddings/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Towards Universal Paraphrastic Sentence Embeddings # Towards Universal Paraphrastic Sentence Embeddings **Authors:** [John Wieting](https://md.research.elamur.ai/free/en/authors/john-wieting/), [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/), [Kevin Gimpel](https://md.research.elamur.ai/free/en/authors/kevin-gimpel/), [Karen Livescu](https://md.research.elamur.ai/free/en/authors/karen-livescu/) · **Year:** 2015 · **Venue:** arXiv:1511.08198 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1511.08198) · [PDF](https://arxiv.org/pdf/1511.08198) · [DOI 10.48550/arxiv.1511.08198](https://doi.org/10.48550/arxiv.1511.08198) · [arXiv 1511.08198](https://arxiv.org/abs/1511.08198) · [OpenAlex](https://openalex.org/W2175723921) **Citations:** 103 citations (openalex-snapshot · 2026-10-14) · [169](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Word embeddings transfer easily across tasks, but comparably reusable sentence vectors were still lacking. This work learns sentence encoders whose cosine similarity reflects how strongly two texts paraphrase each other. The training signal comes from noisy phrase pairs in the Paraphrase Database (PPDB). The authors compare six composition functions of rising complexity: plain word averaging, averaging plus a linear projection, deep averaging networks (DAN), a vanilla RNN, an identity-initialized RNN, and an LSTM. They test on PPDB-like in-domain data and on 22 SemEval textual similarity sets covering news, tweets, forums and captions. The LSTM wins in-domain, but it degrades sharply on other domains. The averaging model, whose only parameters are the word vectors, transfers far better and rivals systems built for the individual tasks. In supervised settings, averaging is again stronger on SICK similarity and entailment, while the LSTM is best on sentiment. The released PARAGRAM-PHRASE XXL embeddings give a cheap, strong baseline for universal sentence representations. ## Key points - Out of domain, word averaging beats the best LSTM by 16.5 Pearson's r on average across the 22 SemEval STS datasets. - Word vectors learned for averaging outperform averaged pretrained vectors: - +17.1 points over GloVe (840B-token Common Crawl); - +12.8 points over PARAGRAM-SL999. - For each SemEval STS competition held between 2012 and 2015, PARAGRAM-PHRASE XXL places within the top 25% of participating entries. It is best or tied for best on 4 datasets. - On the Stanford Sentiment Treebank, the LSTM reaches 89.2% on coarse-grained (binary) classification, which the authors report as a new state of the art. - Using the pretrained embeddings as a prior gives near state-of-the-art results on SICK similarity and entailment. As a black-box feature extractor, they compare favorably with skip-thought vectors (Kiros et al., 2015). Models and code are publicly released. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) ## Cited by Cited by 169 papers of our library cite it (of the 140884 whose bibliographies we read); 21 of them have a page here, 148 not yet. - [Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings](https://md.research.elamur.ai/free/en/papers/bommasani-2020-interpreting-pretrained-contextualized-representations-reductions/) - [Controllable Paraphrase Generation with a Syntactic Exemplar](https://md.research.elamur.ai/free/en/papers/chen-2019-controllable-paraphrase-generation-syntactic-exemplar/) - [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/) - [Probing for semantic evidence of composition by means of simple classification tasks](https://md.research.elamur.ai/free/en/papers/ettinger-2016-probing-semantic-evidence-composition-means/) - [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/) - [Aligning AI With Shared Human Values](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-aligning-ai-shared-human-values/) - [Pretrained Transformers Improve Out-of-Distribution Robustness](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-pretrained-transformers-improve-out-distribution/) - [ParaDetox: Detoxification with Parallel Data](https://md.research.elamur.ai/free/en/papers/logacheva-2022-paradetox-detoxification-parallel-data/) - [An efficient framework for learning sentence representations](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/) - [Paraphrasing Revisited with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/mallinson-2017-paraphrasing-revisited-neural-machine-translation/) - [Learned in Translation: Contextualized Word Vectors](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/) - [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/) - [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://md.research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/) - [DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning](https://md.research.elamur.ai/free/en/papers/popat-2018-declare-debunking-fake-news-false/) - [Learning to Generate Reviews and Discovering Sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/) - [Transfer Learning in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/) - [Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms](https://md.research.elamur.ai/free/en/papers/shen-2018-baseline-needs-more-love-simple/) - [Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition](https://md.research.elamur.ai/free/en/papers/shwartz-2019-still-pain-neck-evaluating-text/) - [A Pragmatics-Centered Evaluation Framework for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/sileo-2022-pragmatics-centered-evaluation-framework-natural/) - [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/en/papers/wieting-2019-beyond-bleu-training-neural-machine/) - [A Joint Sequential and Relational Model for Frame-Semantic Parsing](https://md.research.elamur.ai/free/en/papers/yang-2017-joint-sequential-relational-model-frame/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{wieting2015universal, title = {{Towards Universal Paraphrastic Sentence Embeddings}}, author = {Wieting, John and Bansal, Mohit and Gimpel, Kevin and Livescu, Karen}, year = {2015}, doi = {10.48550/arxiv.1511.08198}, eprint = {1511.08198}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1511.08198} } ``` - [← previous Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/) - [next → Transferring Coreference Resolvers with Posterior Regularization](https://md.research.elamur.ai/free/en/papers/martins-2015-transferring-coreference-resolvers-posterior-regularization/) - [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) ## References 3 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Agirre, Eneko, Diab, Mona, Cer, Daniel, and Gonzalez-Agirre, Aitor. SemEval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation. Association for Computational Linguistics, 2012. [find](https://scholar.google.com/scholar?q=Agirre%2C+Eneko%2C+Diab%2C+Mona%2C+Cer%2C+Daniel%2C+and+Gonzalez%2DAgirre%2C+Aitor%2E+SemEval%2D2012+task+6%3A+A+pilot+on+semantic+textual+similarity%2E+In+Proceedings+of+the+First+Joint+Conference+on+Lexical+and+Computation) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/) 2. Agirre, Eneko, Cer, Daniel, Diab, Mona, Gonzalez-Agirre, Aitor, and Guo, Weiwei. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, 2013. [find](https://scholar.google.com/scholar?q=Agirre%2C+Eneko%2C+Cer%2C+Daniel%2C+Diab%2C+Mona%2C+Gonzalez%2DAgirre%2C+Aitor%2C+and+Guo%2C+Weiwei%2E+%2ASEM+2013+shared+task%3A+Semantic+textual+similarity%2E+In+Second+Joint+Conference+on+Lexical+and+Computational+Semantics+%28) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2013-sem-2013-shared-task-semantic/) 3. Agirre, Eneko, Banea, Carmen, Cardie, Claire, Cer, Daniel, Diab, Mona, Gonzalez-Agirre, Aitor, Guo, Weiwei, Mihalcea, Rada, Rigau, German, and Wiebe, Janyce. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), 2014. [find](https://scholar.google.com/scholar?q=Agirre%2C+Eneko%2C+Banea%2C+Carmen%2C+Cardie%2C+Claire%2C+Cer%2C+Daniel%2C+Diab%2C+Mona%2C+Gonzalez%2DAgirre%2C+Aitor%2C+Guo%2C+Weiwei%2C+Mihalcea%2C+Rada%2C+Rigau%2C+German%2C+and+Wiebe%2C+Janyce%2E+SemEval%2D2014+task+10%3A+Multilingual+semanti) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2014-semeval-2014-task-10-multilingual/)