---
title: "SGT — paper by Shin et al. (2026) — summary and ideas"
description: "The authors argue these act \"off-manifold,\" leaving scattered, prompt-specific patches that attackers can easily undo. 50 references."
authors: ["Sunguk Shin", "Fangzhao Wu", "Byung-Jun Lee", "Meeyoung Cha", "Sungwon Park"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.463/"
doi: "10.18653/v1/2026.acl-long.463"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shin-2026-sgt-securing-open-source-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shin-2026-sgt-securing-open-source-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/) / SGT: Securing Open-Source LLMs Against Malicious Fine-tuning via Safety Guidance Trigger

# SGT: Securing Open-Source LLMs Against Malicious Fine-tuning via Safety Guidance Trigger

**Authors:** [Sunguk Shin](https://md.research.elamur.ai/free/en/authors/sunguk-shin/), [Fangzhao Wu](https://md.research.elamur.ai/free/en/authors/fangzhao-wu/), [Byung-Jun Lee](https://md.research.elamur.ai/free/en/authors/byung-jun-lee/), [Meeyoung Cha](https://md.research.elamur.ai/free/en/authors/meeyoung-cha/) and Sungwon Park · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.463/) · [PDF](https://aclanthology.org/2026.acl-long.463.pdf) · [DOI 10.18653/v1/2026.acl-long.463](https://doi.org/10.18653/v1/2026.acl-long.463) · [OpenAlex](https://openalex.org/W7166886670)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Open-weight LLMs can be stripped of their safety alignment by malicious fine-tuning (MFT) on harmful data. Earlier alignment-stage defenses such as RepNoise and SDD work by adding noise to activations or breaking semantics. The authors argue these act "off-manifold," leaving scattered, prompt-specific patches that attackers can easily undo.

SGT instead sets a coherent safety target. It works in two stages:
- **Stage 1:** With the base model frozen, a single continuous soft trigger is learned in the embedding space. When attached to harmful prompts, it elicits refusals.
- **Stage 2:** The frozen model receives triggered harmful prompts and acts as a teacher, supplying layer-wise "safe" hidden states. A trainable copy then learns to reproduce these states from the same prompts without the trigger.

At inference, no trigger is needed. Experiments use safety data such as WildGuardMix, BeaverTails and Aegis 2.0. The results show that attackers need substantially larger fine-tuning budgets to break SGT models. Representations also stay anchored in a safety region during attacks. This matters for anyone releasing open models that third parties may freely fine-tune.

## Key points
- The soft trigger is one global vector ϕ ∈ R^H. It is regularized with a KL term between outputs for ϕ and a Gaussian-noised ϕ, making the refusal behaviour locally stable.
- The method is compared against Vaccine, RepNoise, SDD and Booster, including on Qwen3-8B-Base.
- SGT shows higher Centered Kernel Alignment (CKA) and k-NN overlap after MFT than the baselines. The authors explain this robustness as "geometric inertia" of a dense safety manifold.
- Benign utility declines slightly on MMLU and OpenBookQA. Follow-up general-purpose SFT recovers part of this loss, but a gap remains.
- SGT offers no guarantee against attackers with unlimited data. The authors frame realistic budgets as roughly 10³–10⁴ examples. Code is available at github.com/ssw1419-korea/SGT under CC BY-NC 4.0.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft Research Asia](https://md.research.elamur.ai/free/en/companies/microsoft-research-asia/) — Company, Beijing, China · 73 works · 60,539 citations
- [Korea Advanced Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/korea-advanced-institute-of-science-and-technology/) — University, Daejeon, South Korea · 51 works · 3,167 citations
- [Korea University](https://md.research.elamur.ai/free/en/universities/korea-university/) — University, Seoul, South Korea · 26 works · 9,490 citations
- [Max Planck Institute for Security and Privacy](https://md.research.elamur.ai/free/en/labs/max-planck-institute-for-security-and-privacy/) — Research institute, Bochum, Germany · 3 works · 499 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Sunguk Shin, Fangzhao Wu, Byung-Jun Lee, Meeyoung Cha and Sungwon Park. 2026. SGT: Securing Open-Source LLMs Against Malicious Fine-tuning via Safety Guidance Trigger. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shin2026sgt,
  title = {{SGT: Securing Open-Source LLMs Against Malicious Fine-tuning via Safety Guidance Trigger}},
  author = {Sunguk Shin, Fangzhao Wu, Byung-Jun Lee, Meeyoung Cha and Sungwon Park},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.463},
  url = {https://doi.org/10.18653/v1/2026.acl-long.463}
}
```

- [← previous SG-UniBuc-NLP at SemEval-2026 Task 6: Multi-Head RoBERTa with Chunking for Long-Context Evasion Detection](https://md.research.elamur.ai/free/en/papers/stefan-2026-sg-unibuc-nlp-semeval-2026/)
- [next → SGVEF-LOOP: Coverage-Guided Progressive Topological Exploration and Fact-Grounded Metamorphic Evaluation for MCP Agents](https://md.research.elamur.ai/free/en/papers/liu-2026-sgvef-loop-coverage-guided-progressive/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/)

## References

50 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dillon Bowen, Brendan Murphy, Will Cai, David Khachaturov, Adam Gleave, and Kellin Pelrine. 2025. Scaling trends for data poisoning in llms. In Proc. of the AAAI Conference on Artificial Intelligence, volume 39, pages 27206–27214. [find](https://scholar.google.com/scholar?q=Dillon+Bowen%2C+Brendan+Murphy%2C+Will+Cai%2C+David+Khachaturov%2C+Adam+Gleave%2C+and+Kellin+Pelrine%2E+2025%2E+Scaling+trends+for+data+poisoning+in+llms%2E+In+Proc%2E+of+the+AAAI+Conference+on+Artificial+Intelligence%2C)
2. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. [Jailbreakbench: An open robustness benchmark for jailbreaking large language models](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/). In…
3. Jinyin Chen, Zhiqi Cao, Ruoxi Chen, Haibin Zheng, Xiao Li, Qi Xuan, and Xing Yang. 2024. Like teacher, like pupil: Transferring backdoors via feature-based knowledge distillation. Computers & Security, 146:104041. [find](https://scholar.google.com/scholar?q=Jinyin+Chen%2C+Zhiqi+Cao%2C+Ruoxi+Chen%2C+Haibin+Zheng%2C+Xiao+Li%2C+Qi+Xuan%2C+and+Xing+Yang%2E+2024%2E+Like+teacher%2C+like+pupil%3A+Transferring+backdoors+via+feature%2Dbased+knowledge+distillation%2E+Computers+%26+Security)
4. Zixuan Chen, Weikai Lu, Xin Lin, and Ziqian Zeng. 2025. [Sdd: Self-degraded defense against malicious fine-tuning](https://md.research.elamur.ai/free/en/papers/chen-2025-sdd-self-degraded-defense-against/). In Proc. of the Association for Computational Linguistics, pages 29109–29125.
5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
6. Aideen Fay, Inés García-Redondo, Qiquan Wang, Haim Dubossarsky, and Anthea Monod. 2025. Holes in latent space: Topological signatures under adversarial influence. arXiv preprint arXiv:2505.20435. — [arXiv:2505.20435](https://arxiv.org/abs/2505.20435) [find](https://scholar.google.com/scholar?q=Aideen+Fay%2C+In%C3%A9s+Garc%C3%ADa%2DRedondo%2C+Qiquan+Wang%2C+Haim+Dubossarsky%2C+and+Anthea+Monod%2E+2025%2E+Holes+in+latent+space%3A+Topological+signatures+under+adversarial+influence%2E+arXiv+preprint+arXiv%3A2505%2E20435%2E+%E2%80%94+%5Ba)
7. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
8. Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien. 2025. [AEGIS2.0: A diverse AI safety dataset and risks taxonomy for alignment of LLM guardrails](https://md.research.elamur.ai/free/en/papers/ghosh-2025-aegis2-0-diverse-ai-safety/). In Proceedings of the 2025 Conference of the Nations of the…
9. Yichen Gong, Delong Ran, Xinlei He, Tianshuo Cong, Anyu Wang, and Xiaoyun Wang. 2025. Safety misalignment against large language models. In Proc. of the Network and Distributed System Security Symposium. [find](https://scholar.google.com/scholar?q=Yichen+Gong%2C+Delong+Ran%2C+Xinlei+He%2C+Tianshuo+Cong%2C+Anyu+Wang%2C+and+Xiaoyun+Wang%2E+2025%2E+Safety+misalignment+against+large+language+models%2E+In+Proc%2E+of+the+Network+and+Distributed+System+Security+Symposi)
10. Danny Halawi, Alexander Wei, Eric Wallace, Tony Tong Wang, Nika Haghtalab, and Jacob Steinhardt. 2024. Covert malicious finetuning: Challenges in safeguarding llm adaptation. In Proc. of the International Conference on Machine Learning, pages 17298–17312. [find](https://scholar.google.com/scholar?q=Danny+Halawi%2C+Alexander+Wei%2C+Eric+Wallace%2C+Tony+Tong+Wang%2C+Nika+Haghtalab%2C+and+Jacob+Steinhardt%2E+2024%2E+Covert+malicious+finetuning%3A+Challenges+in+safeguarding+llm+adaptation%2E+In+Proc%2E+of+the+Internati)
11. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. [Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms](https://md.research.elamur.ai/free/en/papers/han-2024-wildguard-open-one-stop-moderation/). In Proc. of the Neural Information Processing Systems, volume 37, pages 8093–8131.
12. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a. [Aligning ai with shared human values](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-aligning-ai-shared-human-values/). In Proc. of the International Conference on Learning Representations.
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In Proc. of the International Conference on Learning Representations.
14. Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. 2024. Safe lora: The silver lining of reducing safety risks when finetuning large language models. In Proc. of the Neural Information Processing Systems, volume 37, pages 65072–65094. [find](https://scholar.google.com/scholar?q=Chia%2DYi+Hsu%2C+Yu%2DLin+Tsai%2C+Chih%2DHsun+Lin%2C+Pin%2DYu+Chen%2C+Chia%2DMu+Yu%2C+and+Chun%2DYing+Huang%2E+2024%2E+Safe+lora%3A+The+silver+lining+of+reducing+safety+risks+when+finetuning+large+language+models%2E+In+Proc%2E+of+th)
15. Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Joshua Kimball, and Ling Liu. 2025a. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning attack. In Proc. of the International Conference on Machine Learning, pages 25059–25074. [find](https://scholar.google.com/scholar?q=Tiansheng+Huang%2C+Gautam+Bhattacharya%2C+Pratik+Joshi%2C+Joshua+Kimball%2C+and+Ling+Liu%2E+2025a%2E+Antidote%3A+Post%2Dfine%2Dtuning+safety+alignment+for+large+language+models+against+harmful+fine%2Dtuning+attack%2E+In+Pr)
16. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2024a. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. In Proc. of the Neural Information Processing Systems, pages 104521–104555. [find](https://scholar.google.com/scholar?q=Tiansheng+Huang%2C+Sihao+Hu%2C+Fatih+Ilhan%2C+Selim+Furkan+Tekin%2C+and+Ling+Liu%2E+2024a%2E+Lisa%3A+Lazy+safety+alignment+for+large+language+models+against+harmful+fine%2Dtuning+attack%2E+In+Proc%2E+of+the+Neural+Inform)
17. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2025b. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In Proc. of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tiansheng+Huang%2C+Sihao+Hu%2C+Fatih+Ilhan%2C+Selim+Furkan+Tekin%2C+and+Ling+Liu%2E+2025b%2E+Booster%3A+Tackling+harmful+fine%2Dtuning+for+large+language+models+via+attenuating+harmful+perturbation%2E+In+Proc%2E+of+the+I)
18. Tiansheng Huang, Sihao Hu, and Ling Liu. 2024b. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. In Proc. of the Neural Information Processing Systems, volume 37, pages 74058–74088. [find](https://scholar.google.com/scholar?q=Tiansheng+Huang%2C+Sihao+Hu%2C+and+Ling+Liu%2E+2024b%2E+Vaccine%3A+Perturbation%2Daware+alignment+for+large+language+models+against+harmful+fine%2Dtuning+attack%2E+In+Proc%2E+of+the+Neural+Information+Processing+System)
19. IBM Granite Team. 2024. granite-3.1-3b-a800m-base. [https://huggingface.co/ibm-granite/](https://huggingface.co/ibm-granite/) granite-3.1-3b-a800m-base. [↗](https://huggingface.co/ibm-granite/)
20. Jongheon Jeong and Jinwoo Shin. 2020. Consistency regularization for certified robustness of smoothed classifiers. In Proc. of the Neural Information Processing Systems, volume 33, pages 10558–10570. [find](https://scholar.google.com/scholar?q=Jongheon+Jeong+and+Jinwoo+Shin%2E+2020%2E+Consistency+regularization+for+certified+robustness+of+smoothed+classifiers%2E+In+Proc%2E+of+the+Neural+Information+Processing+Systems%2C+volume+33%2C+pages+10558%E2%80%9310570%2E)
21. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. [Beavertails: Towards improved safety alignment of llm via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). In Proc. of the Neural Information Processing Systems, volume 36, pages 24678– 24704…
22. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
23. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. [Similarity of neural network representations revisited](https://md.research.elamur.ai/free/en/papers/kornblith-2019-similarity-neural-network-representations-revisited/). In Proc. of the International Conference on Machine Learning, pages 3519–3529.
24. Hao Li, Lijun Li, Zhenghao Lu, Xianyi Wei, Rui Li, Jing Shao, and Lei Sha. 2025. [Layer-aware representation filtering: Purifying finetuning data to preserve llm safety alignment](https://md.research.elamur.ai/free/en/papers/li-2025-layer-aware-representation-filtering-purifying/). In Proc. of the Empirical Methods in Natural Language Processing, pages 8041–8061.
25. Tianhong Li and Kaiming He. 2025. Back to basics: Let denoising generative models denoise. arXiv preprint arXiv:2511.13720. [find](https://scholar.google.com/scholar?q=Tianhong+Li+and+Kaiming+He%2E+2025%2E+Back+to+basics%3A+Let+denoising+generative+models+denoise%2E+arXiv+preprint+arXiv%3A2511%2E13720%2E)
26. CHEN Liang, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. 2025. Vulnerability-aware alignment: Mitigating uneven forgetting in harmful fine-tuning. In Proc. of the International Conference on Machine Learning, pages 8172–8183. [find](https://scholar.google.com/scholar?q=CHEN+Liang%2C+Xueting+Han%2C+Li+Shen%2C+Jing+Bai%2C+and+Kam%2DFai+Wong%2E+2025%2E+Vulnerability%2Daware+alignment%3A+Mitigating+uneven+forgetting+in+harmful+fine%2Dtuning%2E+In+Proc%2E+of+the+International+Conference+on+Mach)
27. AI @ Meta Llama Team. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
28. Ning Lu, Shengcai Liu, Jiahao Wu, Weiyu Chen, Zhirui Zhang, Yew-Soon Ong, Qi Wang, and Ke Tang. 2025. Safe delta: Consistently preserving safety when fine-tuning llms on diverse datasets. In Proc. of the International Conference on Machine Learning, pages 40537–40559. [find](https://scholar.google.com/scholar?q=Ning+Lu%2C+Shengcai+Liu%2C+Jiahao+Wu%2C+Weiyu+Chen%2C+Zhirui+Zhang%2C+Yew%2DSoon+Ong%2C+Qi+Wang%2C+and+Ke+Tang%2E+2025%2E+Safe+delta%3A+Consistently+preserving+safety+when+fine%2Dtuning+llms+on+diverse+datasets%2E+In+Proc%2E+of+)
29. Wuyuao Mai, Geng Hong, Pei Chen, Xudong Pan, Baojun Liu, Yuan Zhang, Haixin Duan, and Min Yang. 2025. You can’t eat your cake and have it too: The performance degradation of llms with jailbreak defense. In Proc. of the ACM on Web Conference 2025, pages 872–883. [find](https://scholar.google.com/scholar?q=Wuyuao+Mai%2C+Geng+Hong%2C+Pei+Chen%2C+Xudong+Pan%2C+Baojun+Liu%2C+Yuan+Zhang%2C+Haixin+Duan%2C+and+Min+Yang%2E+2025%2E+You+can%E2%80%99t+eat+your+cake+and+have+it+too%3A+The+performance+degradation+of+llms+with+jailbreak+defens)
30. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proc. of the Empirical Methods in Natural Language Processing, pages 2381–2391. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proc%2E+of+the+Empirical+Methods+in+Natu)
31. Sungwon Park, Sungwon Han, Xing Xie, Jae-Gil Lee, and Meeyoung Cha. 2025. Adversarial style augmentation via large language model for robust fake news detection. In Proc. of the ACM on Web Conference 2025, pages 4024–4033. [find](https://scholar.google.com/scholar?q=Sungwon+Park%2C+Sungwon+Han%2C+Xing+Xie%2C+Jae%2DGil+Lee%2C+and+Meeyoung+Cha%2E+2025%2E+Adversarial+style+augmentation+via+large+language+model+for+robust+fake+news+detection%2E+In+Proc%2E+of+the+ACM+on+Web+Conference+)
32. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. [Fine-tuning aligned language models compromises safety, even when users do not intend to!](https://md.research.elamur.ai/free/en/papers/qi-2023-fine-tuning-aligned-language-models/) In Proc. of the International Conference on Learning Representations.
33. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2025. [Smoothllm: Defending large language models against jailbreaking attacks](https://md.research.elamur.ai/free/en/papers/robey-2023-smoothllm-defending-large-language-models/). Transactions on Machine Learning Research.
34. Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. 2024. Representation noising: A defence mechanism against harmful finetuning. In Proc. of the Neural Information Processing Systems, volu… [find](https://scholar.google.com/scholar?q=Domenic+Rosati%2C+Jan+Wehner%2C+Kai+Williams%2C+%C5%81ukasz+Bartoszcze%2C+David+Atanasov%2C+Robie+Gonzales%2C+Subhabrata+Majumdar%2C+Carsten+Maple%2C+Hassan+Sajjad%2C+and+Frank+Rudzicz%2E+2024%2E+Representation+noising%3A+A+defen)
35. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and 1 others. 2024. A strongreject for empty jailbreaks. In Proc. of the Neural Information Processing Systems, volume 37, pages 125416–125440. [find](https://scholar.google.com/scholar?q=Alexandra+Souly%2C+Qingyuan+Lu%2C+Dillon+Bowen%2C+Tu+Trinh%2C+Elvis+Hsieh%2C+Sana+Pandey%2C+Pieter+Abbeel%2C+Justin+Svegliato%2C+Scott+Emmons%2C+Olivia+Watkins%2C+and+1+others%2E+2024%2E+A+strongreject+for+empty+jailbreaks%2E+)
36. Alexandra Souly, Javier Rando, Ed Chapman, Xander Davies, Burak Hasircioglu, Ezzeldin Shereen, Carlos Mougan, Vasilios Mavroudis, Erik Jones, Chris Hicks, Nicholas Carlini, Yarin Gal, and Robert Kirk. 2025. Poisoning attacks on llms require a near-constant number of poison samples. arXiv preprint ar… — [arXiv:2510.07192](https://arxiv.org/abs/2510.07192) [find](https://scholar.google.com/scholar?q=Alexandra+Souly%2C+Javier+Rando%2C+Ed+Chapman%2C+Xander+Davies%2C+Burak+Hasircioglu%2C+Ezzeldin+Shereen%2C+Carlos+Mougan%2C+Vasilios+Mavroudis%2C+Erik+Jones%2C+Chris+Hicks%2C+Nicholas+Carlini%2C+Yarin+Gal%2C+and+Robert+Kirk%2E)
37. Brandon Tran, Jerry Li, and Aleksander Madry. 2018. Spectral signatures in backdoor attacks. In Proc. of the Neural Information Processing Systems, volume 31, pages 8011–8021. [find](https://scholar.google.com/scholar?q=Brandon+Tran%2C+Jerry+Li%2C+and+Aleksander+Madry%2E+2018%2E+Spectral+signatures+in+backdoor+attacks%2E+In+Proc%2E+of+the+Neural+Information+Processing+Systems%2C+volume+31%2C+pages+8011%E2%80%938021%2E)
38. Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, and Chris Koch. 2025. Estimating worst-case frontier risks of open-weight llms. arXiv preprint arXiv:2508.03153. — [arXiv:2508.03153](https://arxiv.org/abs/2508.03153) [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Olivia+Watkins%2C+Miles+Wang%2C+Kai+Chen%2C+and+Chris+Koch%2E+2025%2E+Estimating+worst%2Dcase+frontier+risks+of+open%2Dweight+llms%2E+arXiv+preprint+arXiv%3A2508%2E03153%2E+%E2%80%94+%5BarXiv%3A2508%2E03153%5D%28https%3A%2F%2Farxiv%2E)
39. Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Sharon Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. 2024. Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. In Proc. of the Neural Information Processing Systems, volume 37… [find](https://scholar.google.com/scholar?q=Jiongxiao+Wang%2C+Jiazhao+Li%2C+Yiquan+Li%2C+Xiangyu+Qi%2C+Junjie+Hu%2C+Sharon+Li%2C+Patrick+McDaniel%2C+Muhao+Chen%2C+Bo+Li%2C+and+Chaowei+Xiao%2E+2024%2E+Backdooralign%3A+Mitigating+fine%2Dtuning+based+jailbreak+attack+with+)
40. In Proc. of the Neural Information Processing Systems, volume 36, pages 80079–80110. [find](https://scholar.google.com/scholar?q=In+Proc%2E+of+the+Neural+Information+Processing+Systems%2C+volume+36%2C+pages+80079%E2%80%9380110%2E)
41. Di Wu, Xin Lu, Yanyan Zhao, and Bing Qin. 2025. Separate the wheat from the chaff: A post-hoc approach to safety re-alignment for fine-tuned language models. In Proc. of the Association for Computational Linguistics Findings, pages 1210–1225. [find](https://scholar.google.com/scholar?q=Di+Wu%2C+Xin+Lu%2C+Yanyan+Zhao%2C+and+Bing+Qin%2E+2025%2E+Separate+the+wheat+from+the+chaff%3A+A+post%2Dhoc+approach+to+safety+re%2Dalignment+for+fine%2Dtuned+language+models%2E+In+Proc%2E+of+the+Association+for+Computatio)
42. Nan Yan, Yuqing Li, Xiong Wang, Jing Chen, Kun He, and Bo Li. 2025. Embedx:embedding-based cross-trigger backdoor attack against large language models. In Proc. of the USENIX Security, pages 241–257. [find](https://scholar.google.com/scholar?q=Nan+Yan%2C+Yuqing+Li%2C+Xiong+Wang%2C+Jing+Chen%2C+Kun+He%2C+and+Bo+Li%2E+2025%2E+Embedx%3Aembedding%2Dbased+cross%2Dtrigger+backdoor+attack+against+large+language+models%2E+In+Proc%2E+of+the+USENIX+Security%2C+pages+241%E2%80%93257%2E)
43. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
44. Yifan Yang, Qiao Jin, Furong Huang, and Zhiyong Lu. 2025b. Adversarial prompt and fine-tuning attacks threaten medical large language models. Nature Communications, 16(1):9011. [find](https://scholar.google.com/scholar?q=Yifan+Yang%2C+Qiao+Jin%2C+Furong+Huang%2C+and+Zhiyong+Lu%2E+2025b%2E+Adversarial+prompt+and+fine%2Dtuning+attacks+threaten+medical+large+language+models%2E+Nature+Communications%2C+16%281%29%3A9011%2E)
45. Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2024. On the vulnerability of safety alignment in open-access llms. In Proc. of the Association for Computational Linguistics Findings, pages 9236–9260. [find](https://scholar.google.com/scholar?q=Jingwei+Yi%2C+Rui+Ye%2C+Qisi+Chen%2C+Bin+Zhu%2C+Siheng+Chen%2C+Defu+Lian%2C+Guangzhong+Sun%2C+Xing+Xie%2C+and+Fangzhao+Wu%2E+2024%2E+On+the+vulnerability+of+safety+alignment+in+open%2Daccess+llms%2E+In+Proc%2E+of+the+Associati)
46. Yi Zeng, Weiyu Sun, Tran Huynh, Dawn Song, Bo Li, and Ruoxi Jia. 2024. Beear: Embedding-based adversarial removal of safety backdoors in instruction-tuned language models. In Proc. of the Empirical Methods in Natural Language Processing, pages 13189–13215. [find](https://scholar.google.com/scholar?q=Yi+Zeng%2C+Weiyu+Sun%2C+Tran+Huynh%2C+Dawn+Song%2C+Bo+Li%2C+and+Ruoxi+Jia%2E+2024%2E+Beear%3A+Embedding%2Dbased+adversarial+removal+of+safety+backdoors+in+instruction%2Dtuned+language+models%2E+In+Proc%2E+of+the+Empirical+Me)
47. Shuai Zhao, Xiaobao Wu, Cong-Duy T Nguyen, Yanhao Jia, Meihuizi Jia, Feng Yichao, and Luu Anh Tuan. 2025. Unlearning backdoor attacks for llms with weak-to-strong knowledge distillation. In Proc. of the Association for Computational Linguistics Findings, pages 4937–4952. [find](https://scholar.google.com/scholar?q=Shuai+Zhao%2C+Xiaobao+Wu%2C+Cong%2DDuy+T+Nguyen%2C+Yanhao+Jia%2C+Meihuizi+Jia%2C+Feng+Yichao%2C+and+Luu+Anh+Tuan%2E+2025%2E+Unlearning+backdoor+attacks+for+llms+with+weak%2Dto%2Dstrong+knowledge+distillation%2E+In+Proc%2E+of+t)
48. Huangjie Zheng, Xu Chen, Jiangchao Yao, Hongxia Yang, Chunyuan Li, Ya Zhang, Hao Zhang, Ivor Tsang, Jingren Zhou, and Mingyuan Zhou. 2023. Contrastive attraction and contrastive repulsion for representation learning. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Huangjie+Zheng%2C+Xu+Chen%2C+Jiangchao+Yao%2C+Hongxia+Yang%2C+Chunyuan+Li%2C+Ya+Zhang%2C+Hao+Zhang%2C+Ivor+Tsang%2C+Jingren+Zhou%2C+and+Mingyuan+Zhou%2E+2023%2E+Contrastive+attraction+and+contrastive+repulsion+for+represen)
49. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). arXiv preprint arXiv:2307.15043.
50. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. [Jailbroken: How does llm safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/)

*Updated 1 November 2026*