--- title: "SafeSteer — paper by Zeng et al. (2026) — summary and ideas" description: "Multimodal LLMs are easier to jailbreak than text-only models, especially when the harmful content is hidden in an image. 2 references." authors: ["Xinyi Zeng", "Xue Yang", "Jingyuan Zhang", "Huanqian Yan", "Xiang Chen", "Kaiwen Wei", "Hankun Kang", "Yu Tian"] year: 2026 venue: "Findings of the Association for Computational Linguistics: ACL 2026" source: "https://aclanthology.org/2026.findings-acl.916/" doi: "10.18653/v1/2026.findings-acl.916" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zeng-2026-safesteer-decoding-level-defense-mechanism/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeng-2026-safesteer-decoding-level-defense-mechanism/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/48/) / SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models # SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models **Authors:** [Xinyi Zeng](https://md.research.elamur.ai/free/en/authors/xinyi-zeng/), [Xue Yang](https://md.research.elamur.ai/free/en/authors/xue-yang/), [Jingyuan Zhang](https://md.research.elamur.ai/free/en/authors/jingyuan-zhang/), [Huanqian Yan](https://md.research.elamur.ai/free/en/authors/huanqian-yan/), [Xiang Chen](https://md.research.elamur.ai/free/en/authors/xiang-chen/), [Kaiwen Wei](https://md.research.elamur.ai/free/en/authors/kaiwen-wei/), Hankun Kang and [Yu Tian](https://md.research.elamur.ai/free/en/authors/yu-tian/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.916/) · [PDF](https://aclanthology.org/2026.findings-acl.916.pdf) · [DOI 10.18653/v1/2026.findings-acl.916](https://doi.org/10.18653/v1/2026.findings-acl.916) · [OpenAlex](https://openalex.org/W7166878949) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Multimodal LLMs are easier to jailbreak than text-only models, especially when the harmful content is hidden in an image. Existing defenses either need expensive fine-tuning with annotated data or rewrite inputs and outputs, which adds compute and can hurt helpfulness. The authors first check whether MLLMs (Qwen2.5-VL, Qwen3-VL) internally separate harmful from benign queries. PCA/t-SNE plots of hidden states show a clear split at prefill and during the first few decoding steps. Image-borne attacks sit closer to the benign cluster, which suggests weaker cross-modal safety alignment. Building on this, SafeSteer adds two components at decoding time: - **Decoding-Probe:** a logistic-regression probe on PCA-projected hidden states. It scores candidate tokens for harmfulness and reranks them at each step, so generation is pushed toward safe continuations. - **Modal Semantic Alignment Vector:** derived from the representational gap between image-based and text-based attacks, and injected after prefill. It carries text-side safety over to images. Without fine-tuning, SafeSteer reportedly raises safety by up to 33.4% while preserving utility, with little added overhead. ## Key points - **Analysis dataset:** 200 benign MM-Vet samples plus 200 harmful ones. The harmful set has 50 each of SD, TYPO and SDTYPO from MM-SafetyBench "01-Illegal Activity", and 50 text-only "CB" prompts paired with blank images. - **Evaluation metrics:** - Safety: ASR judged by LlamaGuard-3. - Utility: MM-Vet accuracy scored by GPT-4, and refusal rate on VL-Guard Safe. - Efficiency: training and inference time. - **Baselines:** ECSO, MLLM-Protector and MRD. SafeSteer also generalizes to out-of-domain FigStep and VL-Guard. - **Training cost:** only a linear probe is fitted, with the MLLM frozen; experiments ran on a single NVIDIA A800. - **Ablations:** the Decoding-Probe mainly helps against text-based (CB) attacks, while the alignment vector is key for image-based (SD) attacks. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations - [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations - [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 86 works · 9,097 citations - [Wuhan University](https://md.research.elamur.ai/free/en/universities/wuhan-university/) — University, Wuhan, China · 47 works · 1,246 citations - [Kuaishou](https://md.research.elamur.ai/free/en/companies/kuaishou/) — Company, Beijing, China · 19 works · 95 citations - [Chongqing University](https://md.research.elamur.ai/free/en/universities/chongqing-university/) — University, Chongqing, China · 14 works · 318 citations - [Nanjing University of Aeronautics and Astronautics](https://md.research.elamur.ai/free/en/universities/nanjing-university-of-aeronautics-and-astronautics/) — University, Nanjing, China · 6 works · 2 citations 📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works ## Cite Cite as: Xinyi Zeng, Xue Yang, Jingyuan Zhang, Huanqian Yan, Xiang Chen, Kaiwen Wei, Hankun Kang and Yu Tian. 2026. SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zeng2026safesteer, title = {{SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models}}, author = {Xinyi Zeng, Xue Yang, Jingyuan Zhang, Huanqian Yan, Xiang Chen, Kaiwen Wei, Hankun Kang and Yu Tian}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2026}, year = {2026}, doi = {10.18653/v1/2026.findings-acl.916}, url = {https://doi.org/10.18653/v1/2026.findings-acl.916} } ``` - [← previous SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2026-safemcp-proactive-power-regulation-llm/) - [next → SafetyMem: Adaptive Jailbreak Defense via Dual-Component Safety Memory](https://md.research.elamur.ai/free/en/papers/wang-2026-safetymem-adaptive-jailbreak-defense-dual/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/48/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. [Refusal in language models is mediated by a single direction](https://md.research.elamur.ai/free/en/papers/arditi-2024-refusal-language-models-mediated-single/). Advances in Neural Information Processing Systems, 37:136037–136083. 2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+and+1+others%2E+2022%2E+Training+a+helpful+and+harmless+assistant+wi) *Updated 2 November 2026*