---
title: "Mechanistic Insights into Deferred Semantic Drift in LLMs (2026)"
description: "They propose Deferred Semantic Drift (DSD). 30 references. Decoder-only LLMs cannot revise a token's hidden state after computing it."
authors: ["Jingjie Zeng", "Huayang Li", "Liang Yang", "Shaowu Zhang", "Yuanyuan Sun", "Hongfei Lin"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.57/"
doi: "10.18653/v1/2026.findings-acl.57"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zeng-2026-mechanistic-insights-into-deferred-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeng-2026-mechanistic-insights-into-deferred-semantic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/) / Mechanistic Insights into Deferred Semantic Drift in LLMs

# Mechanistic Insights into Deferred Semantic Drift in LLMs

**Authors:** [Jingjie Zeng](https://md.research.elamur.ai/free/en/authors/jingjie-zeng/), [Huayang Li](https://md.research.elamur.ai/free/en/authors/huayang-li/), [Liang Yang](https://md.research.elamur.ai/free/en/authors/liang-yang/), [Shaowu Zhang](https://md.research.elamur.ai/free/en/authors/shaowu-zhang/), [Yuanyuan Sun](https://md.research.elamur.ai/free/en/authors/yuanyuan-sun/) and [Hongfei Lin](https://md.research.elamur.ai/free/en/authors/hongfei-lin/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.57/) · [PDF](https://aclanthology.org/2026.findings-acl.57.pdf) · [DOI 10.18653/v1/2026.findings-acl.57](https://doi.org/10.18653/v1/2026.findings-acl.57) · [OpenAlex](https://openalex.org/W7166835797)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Decoder-only LLMs cannot revise a token's hidden state after computing it. This raises a question: how do they handle an ambiguous word whose sense becomes clear only from later context? Zeng et al. use metaphors as a testbed, such as "key" meaning a physical tool or a figurative means. They propose Deferred Semantic Drift (DSD). In this account, the ambiguous word keeps a mixed, multi-sense representation. Later context-bearing tokens finish the disambiguation by pulling sense-specific information from its value vector through particular attention heads.

To support this, the authors build a Non-Causal Oracle that lifts the causal mask for a chosen token. It shows the model can separate literal and metaphorical senses when it sees the whole sentence, and it measures the gap that ordinary causal processing must close. They add vector-decoding tools and attention-flow tracing, mainly on Gemma models. Steering the ambiguous token's activations then pushes generations toward literal or figurative readings, which is causal evidence for the mechanism. The work clarifies how meaning is assembled incrementally and offers handles for controlling model behaviour.

## Key points
- The Non-Causal Oracle is a diagnostic probe. It sets the attention mask to zero for one target token at a given layer. The resulting "ideal" state serves as a reference for comparison with the normal causal state.
- Hidden states are interpreted with two tools. The first is a Selfie-inspired "grafting" method, in which a second copy of the model verbalizes an inserted vector. The second is an Orthogonal Difference Explanation, which isolates the semantic shift between two states.
- The steering method is called DDI-ACT. It was tested with coefficients λ ∈ {−2, −1, 0, +1, +2}, and GPT-5 labelled outputs as Prototypical, Metaphorical, or Ambiguous. Negative λ raised the share of literal completions, and positive λ made metaphorical ones dominate.
- The sentence-pair dataset was generated with Gemini 2.5 Pro and checked by human annotators. The data and code are released at github.com/jjtail/dsd.
- Stated limitations: experiments mostly cover the Gemma family. The role of FFN blocks in downstream tokens is left for future work, possibly using SAEs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Dalian University of Technology](https://md.research.elamur.ai/free/en/universities/dalian-university-of-technology/) — University, Dalian, China · 20 works · 46 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 149 works

## Cite

Cite as: Jingjie Zeng, Huayang Li, Liang Yang, Shaowu Zhang, Yuanyuan Sun and Hongfei Lin. 2026. Mechanistic Insights into Deferred Semantic Drift in LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zeng2026mechanistic,
  title = {{Mechanistic Insights into Deferred Semantic Drift in LLMs}},
  author = {Jingjie Zeng, Huayang Li, Liang Yang, Shaowu Zhang, Yuanyuan Sun and Hongfei Lin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.57},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.57}
}
```

- [← previous Measuring Optimal Challenge: Trajectory-Based Difficulty Alignment in Open-Ended Language Tutoring](https://md.research.elamur.ai/free/en/papers/shu-2026-measuring-optimal-challenge-trajectory-based/)
- [next → Mechanistic Interpretability Meets Cognitive Linguistics: Modelling Locative Image Schemas in the Circuit Framework](https://md.research.elamur.ai/free/en/papers/proietti-2026-mechanistic-interpretability-meets-cognitive-linguistics/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/)

## References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others… [find](https://scholar.google.com/scholar?q=Emmanuel+Ameisen%2C+Jack+Lindsey%2C+Adam+Pearce%2C+Wes+Gurnee%2C+Nicholas+L%2E+Turner%2C+Brian+Chen%2C+Craig+Citro%2C+David+Abrahams%2C+Shan+Carter%2C+Basil+Hosmer%2C+Jonathan+Marcus%2C+Michael+Sklar%2C+Adly+Templeton%2C+Trenton)
2. Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. 2024. [Refusal in language models is mediated by a single direction](https://md.research.elamur.ai/free/en/papers/arditi-2024-refusal-language-models-mediated-single/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
3. Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). CoRR, abs/1607.06450.
4. Yonatan Belinkov. 2022. [Probing classifiers: Promises, shortcomings, and advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/). Comput. Linguistics, 48(1):207–219.
5. Haozhe Chen, Carl Vondrick, and Chengzhi Mao. 2024. Selfie: self-interpretation of large language model embeddings. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. [find](https://scholar.google.com/scholar?q=Haozhe+Chen%2C+Carl+Vondrick%2C+and+Chengzhi+Mao%2E+2024%2E+Selfie%3A+self%2Dinterpretation+of+large+language+model+embeddings%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+Learning%2C+ICML%E2%80%9924%2E+JM)
6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arX…
7. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. [Patchscopes: a unifying framework for inspecting hidden representations of language models](https://md.research.elamur.ai/free/en/papers/ghandeharioun-2024-patchscopes-unifying-framework-inspecting-hidden/). In Proceedings of the 41st International Conference on Machine Learning, pages 15466–15490.
8. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. [Finding neurons in a haystack: Case studies with sparse probing](https://md.research.elamur.ai/free/en/papers/gurnee-2023-finding-neurons-haystack-case-studies/).
9. Trans. Mach. Learn. Res., 2023. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2023%2E)
10. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. [What does BERT learn about the structure of language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
11. Walter Kintsch. 2000. Metaphor comprehension: A computational theory. Psychonomic bulletin & review, 7(2):257–266. [find](https://scholar.google.com/scholar?q=Walter+Kintsch%2E+2000%2E+Metaphor+comprehension%3A+A+computational+theory%2E+Psychonomic+bulletin+%26+review%2C+7%282%29%3A257%E2%80%93266%2E)
12. Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022. Discovering latent concepts learned in BERT. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Abdul+Rafae+Khan%2C+Firoj+Alam%2C+Nadir+Durrani%2C+Jia+Xu%2C+and+Hassan+Sajjad%2E+2022%2E+Discovering+latent+concepts+learned+in+BERT%2E+In+International+Conference+on+Learning+Representations%2E)
13. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2021. Incorporating Residual and Normalization Layers into Analysis of Masked Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4547–4568, Online and Punta Cana, Dominica… [find](https://scholar.google.com/scholar?q=Goro+Kobayashi%2C+Tatsuki+Kuribayashi%2C+Sho+Yokoi%2C+and+Kentaro+Inui%2E+2021%2E+Incorporating+Residual+and+Normalization+Layers+into+Analysis+of+Masked+Language+Models%2E+In+Proceedings+of+the+2021+Conference+o)
14. Christiane Fellbaum. 1998. WordNet: An electronic lexical database. MIT press. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998%2E+WordNet%3A+An+electronic+lexical+database%2E+MIT+press%2E)
15. George Lakoff and Mark Johnson. 1980. Metaphors we live by. University of Chicago press. [find](https://scholar.google.com/scholar?q=George+Lakoff+and+Mark+Johnson%2E+1980%2E+Metaphors+we+live+by%2E+University+of+Chicago+press%2E)
16. Sheridan Feucht, David Atkinson, Byron C Wallace, and David Bau. 2024. Token erasure as a footprint of implicit vocabulary items in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9727–9739, Miami, Florida, USA. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Sheridan+Feucht%2C+David+Atkinson%2C+Byron+C+Wallace%2C+and+David+Bau%2E+2024%2E+Token+erasure+as+a+footprint+of+implicit+vocabulary+items+in+LLMs%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+)
17. Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. 2024. Interpreting CLIP’s image representation via text-based decomposition. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yossi+Gandelsman%2C+Alexei+A+Efros%2C+and+Jacob+Steinhardt%2E+2024%2E+Interpreting+CLIP%E2%80%99s+image+representation+via+text%2Dbased+decomposition%2E+In+The+Twelfth+International+Conference+on+Learning+Representations)
18. Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Tom+Dupre+la+Tour%2C+Henk+Tillman%2C+Gabriel+Goh%2C+Rajan+Troll%2C+Alec+Radford%2C+Ilya+Sutskever%2C+Jan+Leike%2C+and+Jeffrey+Wu%2E+2025%2E+Scaling+and+evaluating+sparse+autoencoders%2E+In+The+Thirteenth+Interna)
19. Muhan Gao, TaiMing Lu, Kuai Yu, Adam Byerly, and Daniel Khashabi. 2024. Insights into LLM long-context failures: When transformers know but don‘t tell. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7611–7625, Miami, Florida, USA. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Muhan+Gao%2C+TaiMing+Lu%2C+Kuai+Yu%2C+Adam+Byerly%2C+and+Daniel+Khashabi%2E+2024%2E+Insights+into+LLM+long%2Dcontext+failures%3A+When+transformers+know+but+don%E2%80%98t+tell%2E+In+Findings+of+the+Association+for+Computational)
20. Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others… [find](https://scholar.google.com/scholar?q=Jack+Lindsey%2C+Wes+Gurnee%2C+Emmanuel+Ameisen%2C+Brian+Chen%2C+Adam+Pearce%2C+Nicholas+L%2E+Turner%2C+Craig+Citro%2C+David+Abrahams%2C+Shan+Carter%2C+Basil+Hosmer%2C+Jonathan+Marcus%2C+Michael+Sklar%2C+Adly+Templeton%2C+Trenton)
21. Rowan Hall Maudslay, Simone Teufel, Francis Bond, and James Pustejovsky. 2024. ChainNet: Structured metaphor and metonymy in WordNet. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2984– 2996, Torino… [find](https://scholar.google.com/scholar?q=Rowan+Hall+Maudslay%2C+Simone+Teufel%2C+Francis+Bond%2C+and+James+Pustejovsky%2E+2024%2E+ChainNet%3A+Structured+metaphor+and+metonymy+in+WordNet%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Compu)
22. OpenAI. 2025. Introducing gpt-5. [https://openai.com/index/introducing-gpt-5/](https://openai.com/index/introducing-gpt-5/). [↗](https://openai.com/index/introducing-gpt-5/)
23. Gabriel Peyré and Marco Cuturi. 2019. Computational optimal transport. Found. Trends Mach. Learn., 11(5-6):355–607. [find](https://scholar.google.com/scholar?q=Gabriel+Peyr%C3%A9+and+Marco+Cuturi%2E+2019%2E+Computational+optimal+transport%2E+Found%2E+Trends+Mach%2E+Learn%2E%2C+11%285%2D6%29%3A355%E2%80%93607%2E)
24. Pau Rodriguez, Arno Blaas, Michal Klein, Luca Zappella, Nicholas Apostoloff, marco cuturi, and Xavier Suau. 2025. Controlling language and diffusion models by transporting activations. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Pau+Rodriguez%2C+Arno+Blaas%2C+Michal+Klein%2C+Luca+Zappella%2C+Nicholas+Apostoloff%2C+marco+cuturi%2C+and+Xavier+Suau%2E+2025%2E+Controlling+language+and+diffusion+models+by+transporting+activations%2E+In+The+Thirteen)
25. Kartik K Sreenivasan and Mark D’Esposito. 2019. The what, where and how of delay activity. Nature reviews neuroscience, 20(8):466–481. [find](https://scholar.google.com/scholar?q=Kartik+K+Sreenivasan+and+Mark+D%E2%80%99Esposito%2E+2019%2E+The+what%2C+where+and+how+of+delay+activity%2E+Nature+reviews+neuroscience%2C+20%288%29%3A466%E2%80%93481%2E)
26. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, an… [find](https://scholar.google.com/scholar?q=Adly+Templeton%2C+Tom+Conerly%2C+Jonathan+Marcus%2C+Jack+Lindsey%2C+Trenton+Bricken%2C+Brian+Chen%2C+Adam+Pearce%2C+Craig+Citro%2C+Emmanuel+Ameisen%2C+Andy+Jones%2C+Hoagy+Cunningham%2C+Nicholas+L+Turner%2C+Callum+McDougall%2C+)
27. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics.
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
29. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Jesse+Vig%2C+Sebastian+Gehrmann%2C+Yonatan+Belinkov%2C+Sharon+Qian%2C+Daniel+Nevo%2C+Yaron+Singer%2C+and+Stuart+Shieber%2E+2020%2E+Investigating+gender+bias+in+language+models+using+causal+mediation+analysis%2E+In+Adva)
30. Jingjie Zeng, Zhihao Yang, Qi Yang, Liang Yang, and Hongfei Lin. 2024. Peeling back the layers: Interpreting the storytelling of vit. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7298–7306. [find](https://scholar.google.com/scholar?q=Jingjie+Zeng%2C+Zhihao+Yang%2C+Qi+Yang%2C+Liang+Yang%2C+and+Hongfei+Lin%2E+2024%2E+Peeling+back+the+layers%3A+Interpreting+the+storytelling+of+vit%2E+In+Proceedings+of+the+32nd+ACM+International+Conference+on+Multime)

*Updated 2 November 2026*