---
title: "Scaling Sentence Embeddings with Large Language Models (2024)"
description: "The authors adapt prompt-based representation from PromptBERT and… 86.76 Spearman on STS. 34 references, cited by 67 papers here."
authors: ["Ting Jiang", "Shaohan Huang", "Zhongzhi Luan", "Deqing Wang", "Fuzhen Zhuang"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: EMNLP 2024"
source: "https://aclanthology.org/2024.findings-emnlp.181/"
doi: "10.18653/v1/2024.findings-emnlp.181"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiang-2024-scaling-sentence-embeddings-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiang-2024-scaling-sentence-embeddings-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Scaling Sentence Embeddings with Large Language Models

# Scaling Sentence Embeddings with Large Language Models

**Authors:** Ting Jiang, [Shaohan Huang](https://md.research.elamur.ai/free/en/authors/shaohan-huang/), Zhongzhi Luan, Deqing Wang and Fuzhen Zhuang · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-emnlp.181/) · [PDF](https://aclanthology.org/2024.findings-emnlp.181.pdf) · [DOI 10.18653/v1/2024.findings-emnlp.181](https://doi.org/10.18653/v1/2024.findings-emnlp.181) · [OpenAlex](https://openalex.org/W4404782013)

**Citations:** 59 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/jiang-2024-scaling-sentence-embeddings-large-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks how decoder-only LLMs, which produce text rather than vectors, can yield good sentence embeddings. The authors adapt prompt-based representation from PromptBERT and introduce PromptEOL. In this template the model must describe the input's meaning using a single word, and the hidden state of the next token is taken as the embedding. They argue that BERT's [MASK] template carries an implicit one-word constraint, which unidirectional models lack. Without training, they add in-context demonstrations: sentence–word pairs drawn from STS-B training sentences (labelled by ChatGPT) and from Oxford dictionary definitions, with the best demonstration chosen on the STS-B dev set. With training, they apply contrastive fine-tuning (PromptEOL+CSE, via QLoRA) and an extra alignment step. Experiments cover OPT (125M–66B) and LLaMA (7B–65B) on STS and transfer tasks. Training-free PromptEOL with demonstrations beats unsupervised contrastive models, and fine-tuned versions set a new STS state of the art. Scaling analysis shows that larger models produce more anisotropic embeddings. This explains why STS gains level off without fine-tuning.

## Key points
- Deleting the final period from BERT's prompt template drops STS-B dev Spearman from 73.44 to 33.89. This motivates the explicit one-word constraint in PromptEOL.
- With alignment on top of fine-tuning, the method averages 86.76 Spearman on STS tasks, about 1.8 points above the prior best.
- A fine-tuned 2.7B model with PromptEOL outperforms the 4.8B ST5-Enc.
- On transfer tasks, scores grow with model size, reaching an average of 92.09 for 65B LLaMA. Anisotropy, measured on 100k random Wikipedia sentence pairs, also rises with OPT size.
- Hardware limits capped fine-tuning at 30B parameters with QLoRA. Code and models are at github.com/kongds/scaling_sentemb.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 27 works · 8,089 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang and Fuzhen Zhuang. 2024. Scaling Sentence Embeddings with Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jiang2024scaling,
  title = {{Scaling Sentence Embeddings with Large Language Models}},
  author = {Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang and Fuzhen Zhuang},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.181},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.181}
}
```

- [← previous SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding](https://md.research.elamur.ai/free/en/papers/xu-2024-safedecoding-defending-against-jailbreak-attacks/)
- [next → InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

34 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
2. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. [Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). arXiv preprint arXiv:1708.00055.
3. Qinyuan Cheng, Xiaogui Yang, Tianxiang Sun, Linyang Li, and Xipeng Qiu. 2023. Improving contrastive learning of sentence embeddings from ai feedback. arXiv preprint arXiv:2305.01918. [find](https://scholar.google.com/scholar?q=Qinyuan+Cheng%2C+Xiaogui+Yang%2C+Tianxiang+Sun%2C+Linyang+Li%2C+and+Xipeng+Qiu%2E+2023%2E+Improving+contrastive+learning+of+sentence+embeddings+from+ai+feedback%2E+arXiv+preprint+arXiv%3A2305%2E01918%2E)
4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
5. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljačić, Shang-Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. 2022. [Diffcse: Difference-based contrastive learning for sentence embeddings](https://md.research.elamur.ai/free/en/papers/chuang-2022-diffcse-difference-based-contrastive-learning/). arXiv preprint arXiv:2204.10298.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+2023%2E+Qlora%3A+Efficient+finetuning+of+quantized+llms%2E+arXiv+preprint+arXiv%3A2305%2E14314%2E)
8. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+comparing+the+geometry+of+bert%2C+elmo%2C+and+gpt%2D2+embeddings%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+i)
9. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+2022%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2E)
10. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). arXiv preprint arXiv:2104.08821.
11. Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama. [find](https://scholar.google.com/scholar?q=Xinyang+Geng+and+Hao+Liu%2E+2023%2E+Openllama%3A+An+open+reproduction+of+llama%2E)
12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
13. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
14. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
15. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
16. Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. 2022. Promptbert: Improving bert sentence embeddings with prompts. arXiv preprint arXiv:2201.04337. [find](https://scholar.google.com/scholar?q=Ting+Jiang%2C+Jian+Jiao%2C+Shaohan+Huang%2C+Zihan+Zhang%2C+Deqing+Wang%2C+Fuzhen+Zhuang%2C+Furu+Wei%2C+Haizhen+Huang%2C+Denvy+Deng%2C+and+Qi+Zhang%2E+2022%2E+Promptbert%3A+Improving+bert+sentence+embeddings+with+prompts%2E+arX)
17. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
18. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. [On the sentence embeddings from pre-trained language models](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/). arXiv preprint arXiv:2011.05864.
19. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958.
20. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Computing Surveys, 55(9):1–35.
21. MosaicML. 2023. Introducing mpt-7b: A new standard for open-source, commercially usable llms. Accessed: 2023-05-05. [find](https://scholar.google.com/scholar?q=MosaicML%2E+2023%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2E+Accessed%3A+2023%2D05%2D05%2E)
22. Niklas Muennighoff. 2022. [Sgpt: Gpt sentence embeddings for semantic search](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-sgpt-gpt-sentence-embeddings-semantic/). arXiv preprint arXiv:2202.08904.
23. Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. 2021. [Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models](https://md.research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/). arXiv preprint arXiv:2108.08877.
24. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023b%2E+Llama+2%3A+Open+foundation+and+f)
25. Hayato Tsukagoshi, Ryohei Sasano, and Koichi Takeda. 2021. DefSent: Sentence embeddings using definition sentences. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short P… [find](https://scholar.google.com/scholar?q=Hayato+Tsukagoshi%2C+Ryohei+Sasano%2C+and+Koichi+Takeda%2E+2021%2E+DefSent%3A+Sentence+embeddings+using+definition+sentences%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
26. Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, and Daxin Jiang. 2022a. Pcl: Peer-contrastive learning with diverse augmentations for unsupervised sentence embeddings. arXiv preprint arXiv:2201.12093. [find](https://scholar.google.com/scholar?q=Qiyu+Wu%2C+Chongyang+Tao%2C+Tao+Shen%2C+Can+Xu%2C+Xiubo+Geng%2C+and+Daxin+Jiang%2E+2022a%2E+Pcl%3A+Peer%2Dcontrastive+learning+with+diverse+augmentations+for+unsupervised+sentence+embeddings%2E+arXiv+preprint+arXiv%3A2201%2E)
27. Xing Wu, Chaochen Gao, Zijia Lin, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022b. InfoCSE: Information-aggregated contrastive learning of sentence embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3060–3070, Abu Dhabi, United Arab Emirates. Association f… [find](https://scholar.google.com/scholar?q=Xing+Wu%2C+Chaochen+Gao%2C+Zijia+Lin%2C+Jizhong+Han%2C+Zhongyuan+Wang%2C+and+Songlin+Hu%2E+2022b%2E+InfoCSE%3A+Information%2Daggregated+contrastive+learning+of+sentence+embeddings%2E+In+Findings+of+the+Association+for+Co)
28. Junlei Zhang, Zhenzhong Lan, and Junxian He. 2023. Contrastive learning of sentence embeddings from scratch. arXiv preprint arXiv:2305.15077. [find](https://scholar.google.com/scholar?q=Junlei+Zhang%2C+Zhenzhong+Lan%2C+and+Junxian+He%2E+2023%2E+Contrastive+learning+of+sentence+embeddings+from+scratch%2E+arXiv+preprint+arXiv%3A2305%2E15077%2E)
29. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
30. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290.
31. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). arXiv preprint arXiv:1908.10084.
32. Teven Le Scao, 388 Authors, and Thomas Wolf. 2022. BLOOM: A 176B-parameter open-access multilingual language model. ArXiv, abs/2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+388+Authors%2C+and+Thomas+Wolf%2E+2022%2E+BLOOM%3A+A+176B%2Dparameter+open%2Daccess+multilingual+language+model%2E+ArXiv%2C+abs%2F2211%2E05100%2E)
33. Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Jiarun+Cao%2C+Weijie+Liu%2C+and+Yangyiwen+Ou%2E+2021%2E+Whitening+sentence+representations+for+better+semantics+and+faster+retrieval%2E+arXiv+preprint+arXiv%3A2103%2E15316%2E)
34. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.

*Updated 24 October 2026*