---
title: "Frozen LLMs are Native Decoders for High-Norm Semantic Vectors"
description: "Building on this, the paper proposes a chunking-free long-document compressor. 32.57 EM. 30 references."
authors: ["Yunsheng Zeng", "Yongmei Tan"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1717/"
doi: "10.18653/v1/2026.acl-long.1717"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zeng-2026-frozen-llms-are-native-decoders/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeng-2026-frozen-llms-are-native-decoders/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/) / Frozen LLMs are Native Decoders for High-Norm Semantic Vectors

# Frozen LLMs are Native Decoders for High-Norm Semantic Vectors

**Authors:** Yunsheng Zeng and Yongmei Tan · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1717/) · [PDF](https://aclanthology.org/2026.acl-long.1717.pdf) · [DOI 10.18653/v1/2026.acl-long.1717](https://doi.org/10.18653/v1/2026.acl-long.1717) · [OpenAlex](https://openalex.org/W7166874809)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Soft context-compression methods feed dense vectors to a frozen LLM in place of text, but it has been unclear how such a decoder actually reads these vectors. The authors probe this with a sentence-level reconstruction task. A frozen LLM's last hidden state is passed through a small trainable MLP into a single vector, and the same frozen model must regenerate the original sentence from it. Experiments use Google Conceptual Captions and four frozen decoders. The trained projectors consistently produce vectors whose L2 norm is about 100 times larger than ordinary token embeddings. Shrinking that norm sharply hurts reconstruction, which suggests high magnitude helps make decoding possible. The exact cause (attention dominance, suppression of low-norm inputs, or both) is left open.

Building on this, the paper proposes a chunking-free long-document compressor. Learnable landmark tokens are interleaved with the text and attend bidirectionally over the whole document. Their projected states then replace placeholder embeddings in a frozen Mistral-7B. At 4x compression the method leads on SQuAD and AdversarialQA and is competitive on average across four QA benchmarks.

## Key points
- **Frozen decoders tested:** Mistral-7B-v0.3, Qwen3-8B, Llama-3.2-1B-Instruct and Qwen3-1.7B, covering roughly 1B to 8B parameters. The high-norm pattern appears across these architectures.
- **Attention analysis:** output tokens put heavy attention on the compressed vector, and this focus grows in deeper layers (Layer 0 vs Layer 31 of Mistral).
- **Landmark design:** one ⟨lmk⟩ token is inserted per r text tokens, so r = 4 gives 4x compression.
  - Text tokens use causal masking, while landmarks see the full document.
  - The encoder (Qwen3-Emb) and projector are trained in two stages: reconstruction pretraining, then QA instruction tuning.
- **Ablation:** bidirectional landmark attention clearly beats a causal mask.
  - At 4x: average F1 53.50 vs 32.57, EM 38.35 vs 21.90.
  - At 16x: average F1 45.12 vs 31.61, EM 30.72 vs 20.86.
- **Limitations:** the compression stride is fixed rather than adaptive, evaluation covers only reconstruction and extractive QA, and each QA result comes from a single training run.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 78 works · 33,365 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yunsheng Zeng and Yongmei Tan. 2026. Frozen LLMs are Native Decoders for High-Norm Semantic Vectors. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zeng2026frozen,
  title = {{Frozen LLMs are Native Decoders for High-Norm Semantic Vectors}},
  author = {Yunsheng Zeng and Yongmei Tan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1717},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1717}
}
```

- [← previous From world Englishes to Hyperlingua: English as infrastructural governance in Jordan and the Middle East](https://md.research.elamur.ai/free/en/papers/almashour-2026-world-englishes-hyperlingua-english-infrastructural/)
- [next → FTibSuite: A Comprehensive Resource Suite for Tibetan Vision–Language Modeling](https://md.research.elamur.ai/free/en/papers/xu-2026-ftibsuite-comprehensive-resource-suite-tibetan/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/)

## References

30 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. [Beat the AI: Investigating adversarial human annotation for reading comprehension](https://md.research.elamur.ai/free/en/papers/bartolo-2020-beat-ai-investigating-adversarial-human/). Transactions of the Association for Computational Linguistics, 8:662–678.
2. Haozhe Chen, Carl Vondrick, and Chengzhi Mao. 2024. SelfIE: Self-interpretation of large language model embeddings. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 7373–7388. PMLR. [find](https://scholar.google.com/scholar?q=Haozhe+Chen%2C+Carl+Vondrick%2C+and+Chengzhi+Mao%2E+2024%2E+SelfIE%3A+Self%2Dinterpretation+of+large+language+model+embeddings%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+Learning%2C+volume+235+)
3. Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. 2024. xrag: Extreme context compression for retrieval-augmented generation with one token. In Advances in Neural Information Processing Systems, volume 37, pages 109487–109516. Curran Associates, I… [find](https://scholar.google.com/scholar?q=Xin+Cheng%2C+Xun+Wang%2C+Xingxing+Zhang%2C+Tao+Ge%2C+Si%2DQing+Chen%2C+Furu+Wei%2C+Huishuai+Zhang%2C+and+Dongyan+Zhao%2E+2024%2E+xrag%3A+Extreme+context+compression+for+retrieval%2Daugmented+generation+with+one+token%2E+In+Adv)
4. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. 2023. [Adapting language models to compress contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829–3846, Singapore. Association for Computational Linguistics.
5. Yuhong Dai, Jianxun Lian, Yitian Huang, Wei Zhang, Mingyang Zhou, Mingqi Wu, Xing Xie, and Hao Liao. 2025. Pretraining context compressor for large language models with embedding-based memory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Yuhong+Dai%2C+Jianxun+Lian%2C+Yitian+Huang%2C+Wei+Zhang%2C+Mingyang+Zhou%2C+Mingqi+Wu%2C+Xing+Xie%2C+and+Hao+Liao%2E+2025%2E+Pretraining+context+compressor+for+large+language+models+with+embedding%2Dbased+memory%2E+In+Proc)
6. Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. [In-context autoencoder for context compression in a large language model](https://md.research.elamur.ai/free/en/papers/ge-2023-context-autoencoder-context-compression-large/). In International Conference on Learning Representations.
7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
8. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
9. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). In The Eleventh International Conference on Learning Representations.
10. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. [Atlas: Few-shot learning with retrieval augmented language models](https://md.research.elamur.ai/free/en/papers/izacard-2022-atlas-few-shot-learning-retrieval/). Preprint, arXiv:2208.03299.
11. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
12. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023b. [LLMLingua: Compressing prompts for accelerated inference of large language models](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, Singapore. Association for Comp…
13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
14. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist…
15. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
16. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. [Compressing context to enhance inference efficiency of large language models](https://md.research.elamur.ai/free/en/papers/li-2023-compressing-context-enhance-inference-efficiency/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore. Association for Computational Linguistics.
17. Huanxuan Liao, Wen Hu, Yao Xu, Shizhu He, Jun Zhao, and Kang Liu. 2025. Beyond hard and soft: Hybrid context compression for balancing local and global information retention. Preprint, arXiv:2505.15774. — [arXiv:2505.15774](https://arxiv.org/abs/2505.15774) [find](https://scholar.google.com/scholar?q=Huanxuan+Liao%2C+Wen+Hu%2C+Yao+Xu%2C+Shizhu+He%2C+Jun+Zhao%2C+and+Kang+Liu%2E+2025%2E+Beyond+hard+and+soft%3A+Hybrid+context+compression+for+balancing+local+and+global+information+retention%2E+Preprint%2C+arXiv%3A2505%2E1577)
18. Xiaoqiang Lin, Aritra Ghosh, Bryan Kian Hsiang Low, Anshumali Shrivastava, and Vijai Mohan. 2025. RE-FRAG: Rethinking RAG based decoding. Preprint, arXiv:2509.01092. — [arXiv:2509.01092](https://arxiv.org/abs/2509.01092) [find](https://scholar.google.com/scholar?q=Xiaoqiang+Lin%2C+Aritra+Ghosh%2C+Bryan+Kian+Hsiang+Low%2C+Anshumali+Shrivastava%2C+and+Vijai+Mohan%2E+2025%2E+RE%2DFRAG%3A+Rethinking+RAG+based+decoding%2E+Preprint%2C+arXiv%3A2509%2E01092%2E+%E2%80%94+%5BarXiv%3A2509%2E01092%5D%28https%3A%2F%2Farxiv)
19. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
20. Momose Oyama, Sho Yokoi, and Hidetoshi Shimodaira. 2023. Norm of word embedding encodes information gain. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2108–2130, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Momose+Oyama%2C+Sho+Yokoi%2C+and+Hidetoshi+Shimodaira%2E+2023%2E+Norm+of+word+embedding+encodes+information+gain%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pag)
21. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. [LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression](https://md.research.elamur.ai/free/en/papers/pan-2024-llmlingua-2-data-distillation-efficient/). In Findings of the Associat…
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
23. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proceedings of the 38th Internatio…
24. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling…
25. David Rau, Shuai Wang, Hervé Déjean, and Stéphane Clinchant. 2024. Context embeddings for efficient answer generation in rag. Preprint, arXiv:2407.09252. — [arXiv:2407.09252](https://arxiv.org/abs/2407.09252) [find](https://scholar.google.com/scholar?q=David+Rau%2C+Shuai+Wang%2C+Herv%C3%A9+D%C3%A9jean%2C+and+St%C3%A9phane+Clinchant%2E+2024%2E+Context+embeddings+for+efficient+answer+generation+in+rag%2E+Preprint%2C+arXiv%3A2407%2E09252%2E+%E2%80%94+%5BarXiv%3A2407%2E09252%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F240)
26. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565,…
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
28. Hiroaki Yamagiwa and Hidetoshi Shimodaira. 2025. Norm of mean contextualized embeddings determines their variance. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7778–7808, Abu Dhabi, UAE. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hiroaki+Yamagiwa+and+Hidetoshi+Shimodaira%2E+2025%2E+Norm+of+mean+contextualized+embeddings+determines+their+variance%2E+In+Proceedings+of+the+31st+International+Conference+on+Computational+Linguistics%2C+pag)
29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
30. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…

*Updated 2 November 2026*