---
title: "KV-Embedding — paper by Tang and Yang (2026)"
description: "Decoder-only LLMs can serve as embedding models without fine-tuning, but two architectural issues… 32 references, cited by 1 paper here."
authors: ["Yixuan Tang", "Yi Yang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.540/"
doi: "10.18653/v1/2026.acl-long.540"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2026-kv-embedding-training-free-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2026-kv-embedding-training-free-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/) / KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs

# KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs

**Authors:** [Yixuan Tang](https://md.research.elamur.ai/free/en/authors/yixuan-tang/) and [Yi Yang](https://md.research.elamur.ai/free/en/authors/yi-yang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.540/) · [PDF](https://aclanthology.org/2026.acl-long.540.pdf) · [DOI 10.18653/v1/2026.acl-long.540](https://doi.org/10.18653/v1/2026.acl-long.540) · [OpenAlex](https://openalex.org/W7166867525)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Decoder-only LLMs can serve as embedding models without fine-tuning, but two architectural issues hold them back. Causal masking stops early tokens from seeing later context, and next-token training pushes the last token toward predicting continuations rather than summarizing meaning. Earlier training-free fixes edit the input. Echo duplicates the text, and token prepending adds out-of-vocabulary tokens. KV-Embedding instead works inside the frozen model. At selected layers, it takes the key-value states of the final token, which already summarize the whole sequence, and prepends them as a prefix. Every token can then attend to a global summary in one forward pass. A "compress in one word" prompt counters prediction bias. The layers are chosen automatically where intrinsic dimensionality is lowest, so no manual tuning is needed per architecture. On MTEB with Qwen3-4B, Mistral-7B and Llama-3.1-8B, the method beats prior training-free baselines by up to 10%. It also stays stable on LoCoV1 inputs up to 4,096 tokens. This makes new LLMs usable as embedders cheaply, without contrastive training.

## Key points
- **Baselines and benchmarks:** PromptEOL, Echo and Token Prepending were compared on MTEB and the long-context LoCoV1 benchmark. Simply removing the causal mask (Bi-Attn) made performance collapse.
- **Layer selection:** Choosing layers by intrinsic dimensionality gave the best averages, 0.494 on Qwen3-4B and 0.534 on Mistral-7B. On Mistral, 7 ID-chosen layers outperformed a 10-layer middle-third setup.
- **Embedding-space quality:** On Mistral-7B-Instruct-v0.1, the method had the best alignment (0.6082) and uniformity (−2.3899) of all methods tested.
- **Robustness and default settings:**
  - Five prompt templates scored between 0.529 and 0.540.
  - Hybrid pooling (last-token plus mean) worked most consistently.
  - Re-routing a single KV pair was enough.
- **Scaling and limitations:** With Qwen3-30B-A3B, the average MTEB score reached 0.57, matching the fine-tuned LLM2Vec (LLaMA-2-7B). Latency is higher than plain pooling, and fully supervised embedders still perform better.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 170 works · 26,341 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yixuan Tang and Yi Yang. 2026. KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tang2026kv,
  title = {{KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs}},
  author = {Yixuan Tang and Yi Yang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.540},
  url = {https://doi.org/10.18653/v1/2026.acl-long.540}
}
```

- [← previous KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions](https://md.research.elamur.ai/free/en/papers/wu-2026-knowme-bench-benchmarking-person-understanding/)
- [next → Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge](https://md.research.elamur.ai/free/en/papers/sun-2026-label-effects-shared-heuristic-reliance/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/)

## References

32 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. [LLM2vec: Large language models are secretly powerful text encoders](https://md.research.elamur.ai/free/en/papers/behnamghader-2024-llm2vec-large-language-models-secretly/). In First Conference on Language Modeling.
2. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
3. Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio. 2017. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):12140–8. [find](https://scholar.google.com/scholar?q=Elena+Facco%2C+Maria+d%E2%80%99Errico%2C+Alex+Rodriguez%2C+and+Alessandro+Laio%2E+2017%2E+Estimating+the+intrinsic+dimension+of+datasets+by+a+minimal+neighborhood+information%2E+Scientific+reports%2C+7%281%29%3A12140%E2%80%938%2E)
4. Yuchen Fu, Zifeng Cheng, Zhiwei Jiang, Zhonghui Wang, Yafeng Yin, Zhengliang Li, and Qing Gu. 2025. [Token prepending: A training-free approach for eliciting better sentence embeddings from llms](https://md.research.elamur.ai/free/en/papers/fu-2025-token-prepending-training-free-approach/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon…
5. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pages 5484–5495. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+2021%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
6. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
7. Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2024. [Scaling sentence embeddings with large language models](https://md.research.elamur.ai/free/en/papers/jiang-2024-scaling-sentence-embeddings-large-language/). In Findings of the Association for Computational Linguistics: EMNLP, pages 3182–3196. Association for Computational Linguistics.
8. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research,… [find](https://scholar.google.com/scholar?q=Angelos+Katharopoulos%2C+Apoorv+Vyas%2C+Nikolaos+Pappas%2C+and+Fran%C3%A7ois+Fleuret%2E+2020%2E+Transformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2E+In+Proceedings+of+the+37th+International)
9. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. [Nv-embed: Improved techniques for training llms as generalist embedding models](https://md.research.elamur.ai/free/en/papers/lee-2024-nv-embed-improved-techniques-training/). In The Thirteenth International Conference on Learning Representations, ICLR 2025. OpenReview.net.
10. Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. 2024. Meta-task prompting elicits embeddings from large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, pages 10141– 10157.… [find](https://scholar.google.com/scholar?q=Yibin+Lei%2C+Di+Wu%2C+Tianyi+Zhou%2C+Tao+Shen%2C+Yu+Cao%2C+Chongyang+Tao%2C+and+Andrew+Yates%2E+2024%2E+Meta%2Dtask+prompting+elicits+embeddings+from+large+language+models%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+)
11. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented generation for knowledge-intensive NLP tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Advances in Neural Information Proc…
12. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
13. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. [Towards general text embeddings with multi-stage contrastive learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/). CoRR, abs/2308.03281.
14. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
15. Niklas Muennighoff. 2022. [SGPT: GPT sentence embeddings for semantic search](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-sgpt-gpt-sentence-embeddings-semantic/). CoRR, abs/2202.08904.
16. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2025. [Generative representational instruction tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/). In The Thirteenth International Conference on Learning Representations, ICLR 2025. OpenReview.net.
17. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. [MTEB: Massive text embedding benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/). In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037. Association for Computational Linguistics.
18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+and+1+others%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
19. Jon Saad-Falcon, Daniel Y. Fu, Simran Arora, Neel Guha, and Christopher Ré. 2024. Benchmarking and building long-context retrieval models with loco and m2-bert. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. [find](https://scholar.google.com/scholar?q=Jon+Saad%2DFalcon%2C+Daniel+Y%2E+Fu%2C+Simran+Arora%2C+Neel+Guha%2C+and+Christopher+R%C3%A9%2E+2024%2E+Benchmarking+and+building+long%2Dcontext+retrieval+models+with+loco+and+m2%2Dbert%2E+In+Proceedings+of+the+41st+Internationa)
20. Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. 2025. [Layer by layer: Uncovering hidden representations in language models](https://md.research.elamur.ai/free/en/papers/skean-2025-layer-layer-uncovering-hidden-representations/). In Forty-second International Conference on Machine Learning.
21. Tingyu Song, Guo Gan, Mingsheng Shang, and Yilun Zhao. 2025. IFIR: A comprehensive benchmark for evaluating instruction-following in expert-domain information retrieval. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hum… [find](https://scholar.google.com/scholar?q=Tingyu+Song%2C+Guo+Gan%2C+Mingsheng+Shang%2C+and+Yilun+Zhao%2E+2025%2E+IFIR%3A+A+comprehensive+benchmark+for+evaluating+instruction%2Dfollowing+in+expert%2Ddomain+information+retrieval%2E+In+Proceedings+of+the+2025+Con)
22. Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. 2025. [Repetition improves language model embeddings](https://md.research.elamur.ai/free/en/papers/springer-2024-repetition-improves-language-model-embeddings/). In The Thirteenth International Conference on Learning Representations, ICLR 2025. OpenReview.net.
23. Raghuveer Thirukovalluru and Bhuwan Dhingra. 2025. GenEOL: Harnessing the generative power of LLMs for training-free sentence embeddings. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2295–2308, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Raghuveer+Thirukovalluru+and+Bhuwan+Dhingra%2E+2025%2E+GenEOL%3A+Harnessing+the+generative+power+of+LLMs+for+training%2Dfree+sentence+embeddings%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+)
24. Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. 2023. The geometry of hidden representations of large transformer models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems… [find](https://scholar.google.com/scholar?q=Lucrezia+Valeriani%2C+Diego+Doimo%2C+Francesca+Cuturello%2C+Alessandro+Laio%2C+Alessio+Ansuini%2C+and+Alberto+Cazzaniga%2E+2023%2E+The+geometry+of+hidden+representations+of+large+transformer+models%2E+In+Advances+in+)
25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998– 6008.
26. Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. [Label words are anchors: An information flow perspective for understanding in-context learning](https://md.research.elamur.ai/free/en/papers/wang-2023-label-words-anchors-information-flow/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 98…
27. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. [Improving text embeddings with large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, pages 11897–11916. Association f…
28. Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research, pages 9929–9939. P… [find](https://scholar.google.com/scholar?q=Tongzhou+Wang+and+Phillip+Isola%2E+2020%2E+Understanding+contrastive+representation+learning+through+alignment+and+uniformity+on+the+hypersphere%2E+In+Proceedings+of+the+37th+International+Conference+on+Mac)
29. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+International+Conference+on+Learning+Representations)
30. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
31. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Pr… [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+W%2E+Barrett%2C+Zhangyang+Wang%2C+and+Beidi+Chen%2E+2023%2E+H2O%3A+heavy%2Dhitter+orac)
32. Ziyin Zhang, Zihan Liao, Hang Yu, Peng Di, and Rui Wang. 2025. F2LLM technical report: Matching SOTA embedding performance with 6 million open-source data. CoRR, abs/2510.02294. — [arXiv:2510.02294](https://arxiv.org/abs/2510.02294) [find](https://scholar.google.com/scholar?q=Ziyin+Zhang%2C+Zihan+Liao%2C+Hang+Yu%2C+Peng+Di%2C+and+Rui+Wang%2E+2025%2E+F2LLM+technical+report%3A+Matching+SOTA+embedding+performance+with+6+million+open%2Dsource+data%2E+CoRR%2C+abs%2F2510%2E02294%2E+%E2%80%94+%5BarXiv%3A2510%2E02294%5D%28h)

*Updated 2 November 2026*