---
title: "SDD: Self-Degraded Defense against Malicious Fine-tuning (2025)"
description: "Open-weight LLMs that have been safety-aligned can be turned harmful again by fine-tuning them on a… 60 references, cited by 4 papers here."
authors: ["ZiXuan Chen", "Weikai Lu", "Xin Lin", "Ziqian Zeng"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1412/"
doi: "10.18653/v1/2025.acl-long.1412"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-sdd-self-degraded-defense-against/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-sdd-self-degraded-defense-against/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/) / SDD: Self-Degraded Defense against Malicious Fine-tuning

# SDD: Self-Degraded Defense against Malicious Fine-tuning

**Authors:** [ZiXuan Chen](https://md.research.elamur.ai/free/en/authors/zixuan-chen/), Weikai Lu, Xin Lin and [Ziqian Zeng](https://md.research.elamur.ai/free/en/authors/ziqian-zeng/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1412/) · [PDF](https://aclanthology.org/2025.acl-long.1412.pdf) · [DOI 10.18653/v1/2025.acl-long.1412](https://doi.org/10.18653/v1/2025.acl-long.1412) · [OpenAlex](https://openalex.org/W4412889791)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/chen-2025-sdd-self-degraded-defense-against/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open-weight LLMs that have been safety-aligned can be turned harmful again by fine-tuning them on a small amount of harmful data (malicious fine-tuning, MFT). Most existing defenses rest on empirical observations. The authors model an LLM as a feature selector plus classifier, using invariant and spurious features in the style of Lin et al., 2023. With this framework, they explain why standard alignment fails against MFT. They also show that, under certain conditions, MFT can be made to destroy a model's general abilities.

Their method, Self-Degraded Defense (SDD), uses a simple supervised fine-tuning step. Harmful prompts are paired with competent but unrelated benign answers, for example coffee-making instructions given in reply to a violent request. Any later MFT then pushes down the probability of these well-formed outputs, and the model's overall capability collapses. A degraded model cannot usefully carry out harmful requests. SDD can be added at any training stage and keeps the model usable under benign fine-tuning. This gives open-model releasers a pre-release safeguard that does not depend on controlling what attackers do.

## Key points
- SDD's training set uses only 500 harmful queries from AdvBench, each matched with a high-quality response on an unrelated topic.
- Attacks were evaluated with LLM-Finetune-Safety (capped at 100 malicious samples) and BeaverTails-Evaluation (larger). On the larger set, SDD on Llama2-7b-chat kept its protection even when the attack data was 20 times the size of its own training data. The unprotected Vanilla model did not hold up.
- SDD_reject is a variant that prepends a fixed refusal statement to the training answers. It matches SDD's harmlessness and achieves a higher explicit rejection rate, as judged by GPT-4.
- The defense also worked on other backbones: Phi-2 (2.7B) and GLM-3 (6B).
- Code is available at github.com/ZeroNLP/SDD. A noted limitation is that replying with irrelevant content, even with a refusal prefix, does not match how humans naturally decline requests.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [South China University of Technology](https://md.research.elamur.ai/free/en/universities/south-china-university-of-technology/) — University, Guangzhou, China · 22 works · 1,554 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: ZiXuan Chen, Weikai Lu, Xin Lin and Ziqian Zeng. 2025. SDD: Self-Degraded Defense against Malicious Fine-tuning. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2025sdd,
  title = {{SDD: Self-Degraded Defense against Malicious Fine-tuning}},
  author = {ZiXuan Chen, Weikai Lu, Xin Lin and Ziqian Zeng},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1412},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1412}
}
```

- [← previous SciClaims: An End-to-End Generative System for Biomedical Claim Analysis](https://md.research.elamur.ai/free/en/papers/ortega-2025-sciclaims-end-end-generative-system/)
- [next → Sentiment Analysis on Arabic Dialects: A Multi-Dialect Benchmark](https://md.research.elamur.ai/free/en/papers/nwesri-2025-sentiment-analysis-arabic-dialects-multi/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/12/)

## References

60 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816. [find](https://scholar.google.com/scholar?q=Zeyuan+Allen%2DZhu+and+Yuanzhi+Li%2E+2020%2E+Towards+understanding+ensemble%2C+knowledge+distillation+and+self%2Ddistillation+in+deep+learning%2E+arXiv+preprint+arXiv%3A2012%2E09816%2E)
3. Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
4. Anthropic. 2023. Claude. [https://claude.ai/](https://claude.ai/). [↗](https://claude.ai/)
5. Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. [Invariant risk minimization](https://md.research.elamur.ai/free/en/papers/arjovsky-2019-invariant-risk-minimization/). arXiv preprint arXiv:1907.02893.
6. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam Mc-Candlish, Chri… [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Benjamin+Mann%2C+Nova+DasSarma%2C+Nelson+Elhage%2C+Zac+Hatfield%2DDodds%2C+Danny+Hernandez%2C+Jackson+Ker)
7. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
8. Rishabh Bhardwaj and Soujanya Poria. 2023. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. CoRR, abs/2310.14303. — [arXiv:2310.14303](https://arxiv.org/abs/2310.14303) [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj+and+Soujanya+Poria%2E+2023%2E+Language+model+unalignment%3A+Parametric+red%2Dteaming+to+expose+hidden+harms+and+biases%2E+CoRR%2C+abs%2F2310%2E14303%2E+%E2%80%94+%5BarXiv%3A2310%2E14303%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E1)
9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
10. Yangyi Chen, Hongcheng Gao, Ganqu Cui, Fanchao Qi, Longtao Huang, Zhiyuan Liu, and Maosong Sun. 2022. Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial NLP. In EMNLP, pages 11222–11237. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yangyi+Chen%2C+Hongcheng+Gao%2C+Ganqu+Cui%2C+Fanchao+Qi%2C+Longtao+Huang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2022%2E+Why+should+adversarial+perturbations+be+imperceptible%3F+rethink+the+research+paradigm+in+adversaria)
11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu,… — [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/)
12. Pranav Gade, Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023. Badllama: cheaply removing safety fine-tuning from llama 2-chat 13b. CoRR, abs/2311.00117. — [arXiv:2311.00117](https://arxiv.org/abs/2311.00117) [find](https://scholar.google.com/scholar?q=Pranav+Gade%2C+Simon+Lermen%2C+Charlie+Rogers%2DSmith%2C+and+Jeffrey+Ladish%2E+2023%2E+Badllama%3A+cheaply+removing+safety+fine%2Dtuning+from+llama+2%2Dchat+13b%2E+CoRR%2C+abs%2F2311%2E00117%2E+%E2%80%94+%5BarXiv%3A2311%2E00117%5D%28https%3A%2F%2Farxiv)
13. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. [Chatglm: A family of large language models from glm-130b to glm-4 all tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/). arXiv preprint arXiv:2406.12793.
14. Anjali Gopal, Nathan Helm-Burger, Lenni Justen, Emily H Soice, Tiffany Tzeng, Geetha Jeyapragasan, Simon Grimm, Benjamin Mueller, and Kevin M Esvelt. 2023. Will releasing the weights of large language models grant widespread access to pandemic agents? arXiv preprint arXiv:2310.18233. — [arXiv:2310.18233](https://arxiv.org/abs/2310.18233) [find](https://scholar.google.com/scholar?q=Anjali+Gopal%2C+Nathan+Helm%2DBurger%2C+Lenni+Justen%2C+Emily+H+Soice%2C+Tiffany+Tzeng%2C+Geetha+Jeyapragasan%2C+Simon+Grimm%2C+Benjamin+Mueller%2C+and+Kevin+M+Esvelt%2E+2023%2E+Will+releasing+the+weights+of+large+language)
15. Julian Hazell. 2023. Large language models can be used to effectively scale spear phishing campaigns. CoRR, abs/2305.06972. — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) [find](https://scholar.google.com/scholar?q=Julian+Hazell%2E+2023%2E+Large+language+models+can+be+used+to+effectively+scale+spear+phishing+campaigns%2E+CoRR%2C+abs%2F2305%2E06972%2E+%E2%80%94+%5BarXiv%3A2305%2E06972%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E06972%29)
16. Peter Henderson, Eric Mitchell, Christopher D. Manning, Dan Jurafsky, and Chelsea Finn. 2023. Self-destructing models: Increasing the costs of harmful dual uses of foundation models. In AIES, pages 287– 296. ACM. [find](https://scholar.google.com/scholar?q=Peter+Henderson%2C+Eric+Mitchell%2C+Christopher+D%2E+Manning%2C+Dan+Jurafsky%2C+and+Chelsea+Finn%2E+2023%2E+Self%2Ddestructing+models%3A+Increasing+the+costs+of+harmful+dual+uses+of+foundation+models%2E+In+AIES%2C+pages+28)
17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In ICLR. OpenReview.net.
18. Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. 2023. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. CoRR, abs/2305.11176. — [arXiv:2305.11176](https://arxiv.org/abs/2305.11176) [find](https://scholar.google.com/scholar?q=Siyuan+Huang%2C+Zhengkai+Jiang%2C+Hao+Dong%2C+Yu+Qiao%2C+Peng+Gao%2C+and+Hongsheng+Li%2E+2023%2E+Instruct2act%3A+Mapping+multi%2Dmodality+instructions+to+robotic+actions+with+large+language+model%2E+CoRR%2C+abs%2F2305%2E11176%2E)
19. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2024a. Harmful fine-tuning attacks and defenses for [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2409.18169. — [arXiv:2409.18169](https://arxiv.org/abs/2409.18169)
20. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2025. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In ICLR. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tiansheng+Huang%2C+Sihao+Hu%2C+Fatih+Ilhan%2C+Selim+Furkan+Tekin%2C+and+Ling+Liu%2E+2025%2E+Booster%3A+Tackling+harmful+fine%2Dtuning+for+large+language+models+via+attenuating+harmful+perturbation%2E+In+ICLR%2E+OpenRevie)
21. Tiansheng Huang, Sihao Hu, and Ling Liu. 2024b. Vaccine: Perturbation-aware alignment for large language model. CoRR, abs/2402.01109. — [arXiv:2402.01109](https://arxiv.org/abs/2402.01109) [find](https://scholar.google.com/scholar?q=Tiansheng+Huang%2C+Sihao+Hu%2C+and+Ling+Liu%2E+2024b%2E+Vaccine%3A+Perturbation%2Daware+alignment+for+large+language+model%2E+CoRR%2C+abs%2F2402%2E01109%2E+%E2%80%94+%5BarXiv%3A2402%2E01109%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E01109%29)
22. Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3. [find](https://scholar.google.com/scholar?q=Mojan+Javaheripi%2C+S%C3%A9bastien+Bubeck%2C+Marah+Abdin%2C+Jyoti+Aneja%2C+Sebastien+Bubeck%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Weizhu+Chen%2C+Allie+Del+Giorno%2C+Ronen+Eldan%2C+Sivakanth+Gopi%2C+et+al%2E+2023%2E+Phi%2D2%3A+The+surprisin)
23. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. [Beavertails: Towards improved safety alignment of LLM via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). In NeurIPS.
24. Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. CoRR, abs/2310.20624. [find](https://scholar.google.com/scholar?q=Simon+Lermen%2C+Charlie+Rogers%2DSmith%2C+and+Jeffrey+Ladish%2E+2023%2E+Lora+fine%2Dtuning+efficiently+undoes+safety+training+in+llama+2%2Dchat+70b%2E+CoRR%2C+abs%2F2310%2E20624%2E)
25. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023. Rain: Your language models can align themselves without finetuning. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuhui+Li%2C+Fangyun+Wei%2C+Jinjing+Zhao%2C+Chao+Zhang%2C+and+Hongyang+Zhang%2E+2023%2E+Rain%3A+Your+language+models+can+align+themselves+without+finetuning%2E+In+The+Twelfth+International+Conference+on+Learning+Repre)
26. Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. 2023. Spurious feature diversification improves out-of-distribution generalization. Preprint, arXiv:2309.17230. — [arXiv:2309.17230](https://arxiv.org/abs/2309.17230) [find](https://scholar.google.com/scholar?q=Yong+Lin%2C+Lu+Tan%2C+Yifan+Hao%2C+Honam+Wong%2C+Hanze+Dong%2C+Weizhong+Zhang%2C+Yujiu+Yang%2C+and+Tong+Zhang%2E+2023%2E+Spurious+feature+diversification+improves+out%2Dof%2Ddistribution+generalization%2E+Preprint%2C+arXiv%3A230)
27. Guozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo, Qi Mu, and Li Shen. 2024a. Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturbation. CoRR, abs/2410.09760. [find](https://scholar.google.com/scholar?q=Guozhi+Liu%2C+Weiwei+Lin%2C+Tiansheng+Huang%2C+Ruichao+Mo%2C+Qi+Mu%2C+and+Li+Shen%2E+2024a%2E+Targeted+vaccine%3A+Safety+alignment+for+large+language+models+against+harmful+fine%2Dtuning+via+layer%2Dwise+perturbation%2E+Co)
28. Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. 2024b. Robustifying safety-aligned large language models through clean data curation. arXiv preprint arXiv:2405.19358. — [arXiv:2405.19358](https://arxiv.org/abs/2405.19358) [find](https://scholar.google.com/scholar?q=Xiaoqun+Liu%2C+Jiacheng+Liang%2C+Muchao+Ye%2C+and+Zhaohan+Xi%2E+2024b%2E+Robustifying+safety%2Daligned+large+language+models+through+clean+data+curation%2E+arXiv+preprint+arXiv%3A2405%2E19358%2E+%E2%80%94+%5BarXiv%3A2405%2E19358%5D%28http)
29. Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. 2023. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. CoRR, abs/2308.09442. — [arXiv:2308.09442](https://arxiv.org/abs/2308.09442) [find](https://scholar.google.com/scholar?q=Yizhen+Luo%2C+Jiahuan+Zhang%2C+Siqi+Fan%2C+Kai+Yang%2C+Yushuai+Wu%2C+Mu+Qiao%2C+and+Zaiqing+Nie%2E+2023%2E+Biomedgpt%3A+Open+multimodal+generative+pre%2Dtrained+transformer+for+biomedicine%2E+CoRR%2C+abs%2F2308%2E09442%2E+%E2%80%94+%5BarXiv)
30. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024a. Simpo: Simple preference optimization with a reference-free reward. Preprint, arXiv:2405.14734. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+2024a%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+Preprint%2C+arXiv%3A2405%2E14734%2E)
31. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024b. Simpo: Simple preference optimization with a reference-free reward. In NeurIPS. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+2024b%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+In+NeurIPS%2E)
32. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+EMNLP%2E)
33. Seumas Miller and Michael J Selgelid. 2007. Ethical and philosophical consideration of the dual-use dilemma in the biological sciences. Science and engineering ethics, 13:523–580. [find](https://scholar.google.com/scholar?q=Seumas+Miller+and+Michael+J+Selgelid%2E+2007%2E+Ethical+and+philosophical+consideration+of+the+dual%2Duse+dilemma+in+the+biological+sciences%2E+Science+and+engineering+ethics%2C+13%3A523%E2%80%93580%2E)
34. OpenAI. 2022. Introducing ChatGPT. openai.com/blog/chatgpt/. https:// [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Introducing+ChatGPT%2E+openai%2Ecom%2Fblog%2Fchatgpt%2F%2E+https%3A%2F%2F)
35. OpenAI. n.d. Sharegpt. Accessed: 2025-02-16. [find](https://scholar.google.com/scholar?q=OpenAI%2E+n%2Ed%2E+Sharegpt%2E+Accessed%3A+2025%2D02%2D16%2E)
36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
37. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2024. Safety alignment should be made more than just a few tokens deep. Preprint, arXiv:2406.05946. [find](https://scholar.google.com/scholar?q=Xiangyu+Qi%2C+Ashwinee+Panda%2C+Kaifeng+Lyu%2C+Xiao+Ma%2C+Subhrajit+Roy%2C+Ahmad+Beirami%2C+Prateek+Mittal%2C+and+Peter+Henderson%2E+2024%2E+Safety+alignment+should+be+made+more+than+just+a+few+tokens+deep%2E+Preprint%2C+a)
38. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. [Fine-tuning aligned language models compromises safety, even when users do not intend to!](https://md.research.elamur.ai/free/en/papers/qi-2023-fine-tuning-aligned-language-models/) CoRR, abs/2310.03693.
39. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In NeurIPS.
40. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). arXiv preprint arXiv:1908.10084.
41. Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, Subhabrata Majumdar, Hassan Sajjad, Frank Rudzicz, et al. 2024. Representation noising: A defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems, 37:12636–12676. [find](https://scholar.google.com/scholar?q=Domenic+Rosati%2C+Jan+Wehner%2C+Kai+Williams%2C+Lukasz+Bartoszcze%2C+Robie+Gonzales%2C+Subhabrata+Majumdar%2C+Hassan+Sajjad%2C+Frank+Rudzicz%2C+et+al%2E+2024%2E+Representation+noising%3A+A+defence+mechanism+against+harmful)
42. Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. 2020. The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761. — [arXiv:2010.05761](https://arxiv.org/abs/2010.05761) [find](https://scholar.google.com/scholar?q=Elan+Rosenfeld%2C+Pradeep+Ravikumar%2C+and+Andrej+Risteski%2E+2020%2E+The+risks+of+invariant+risk+minimization%2E+arXiv+preprint+arXiv%3A2010%2E05761%2E+%E2%80%94+%5BarXiv%3A2010%2E05761%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E05761%29)
43. Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. 2023. On the exploitability of instruction tuning. In NeurIPS. [find](https://scholar.google.com/scholar?q=Manli+Shu%2C+Jiongxiao+Wang%2C+Chen+Zhu%2C+Jonas+Geiping%2C+Chaowei+Xiao%2C+and+Tom+Goldstein%2E+2023%2E+On+the+exploitability+of+instruction+tuning%2E+In+NeurIPS%2E)
44. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. In NeurIPS. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F%2E+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+In+NeurIPS%2E)
45. Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, and Mantas Mazeika. 2025. Tamper-resistant safeguards for open-weight llms. In ICLR. OpenReview.net. [find](https://scholar.google.com/scholar?q=Rishub+Tamirisa%2C+Bhrugu+Bharathi%2C+Long+Phan%2C+Andy+Zhou%2C+Alice+Gatti%2C+Tarun+Suresh%2C+Maxwell+Lin%2C+Justin+Wang%2C+Rowan+Wang%2C+Ron+Arel%2C+Andy+Zou%2C+Dawn+Song%2C+Bo+Li%2C+Dan+Hendrycks%2C+and+Mantas+Mazeika%2E+2025%2E+)
46. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
47. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, a…
48. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023b. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, a…
49. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023c. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
50. Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. 2023a. Rlcd: Reinforcement learning from contrastive distillation for lm alignment. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Yang%2C+Dan+Klein%2C+Asli+Celikyilmaz%2C+Nanyun+Peng%2C+and+Yuandong+Tian%2E+2023a%2E+Rlcd%3A+Reinforcement+learning+from+contrastive+distillation+for+lm+alignment%2E+In+The+Twelfth+International+Conference+on+)
51. Xianjun Yang, Xiao Wang, Qi Zhang, Linda R. Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023b. Shadow alignment: The ease of subverting safely-aligned language models. CoRR, abs/2310.02949. [find](https://scholar.google.com/scholar?q=Xianjun+Yang%2C+Xiao+Wang%2C+Qi+Zhang%2C+Linda+R%2E+Petzold%2C+William+Yang+Wang%2C+Xun+Zhao%2C+and+Dahua+Lin%2E+2023b%2E+Shadow+alignment%3A+The+ease+of+subverting+safely%2Daligned+language+models%2E+CoRR%2C+abs%2F2310%2E02949%2E)
52. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. CoRR, abs/2401.10020. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E+CoRR%2C+abs%2F2401%2E10020%2E)
53. Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. 2023. Removing RLHF protections in GPT-4 via fine-tuning. CoRR, abs/2311.05553. [find](https://scholar.google.com/scholar?q=Qiusi+Zhan%2C+Richard+Fang%2C+Rohan+Bindu%2C+Akul+Gupta%2C+Tatsunori+Hashimoto%2C+and+Daniel+Kang%2E+2023%2E+Removing+RLHF+protections+in+GPT%2D4+via+fine%2Dtuning%2E+CoRR%2C+abs%2F2311%2E05553%2E)
54. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023a. LIMA: less is more for alignment. In NeurIPS. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srinivasan+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+Susan+Zhang%2C+Gargi+Ghosh%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+2023a%2E+LIMA)
55. Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023b. Making harmful behaviors unlearnable for large language models. CoRR, abs/2311.02105. [find](https://scholar.google.com/scholar?q=Xin+Zhou%2C+Yi+Lu%2C+Ruotian+Ma%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2023b%2E+Making+harmful+behaviors+unlearnable+for+large+language+models%2E+CoRR%2C+abs%2F2311%2E02105%2E)
56. Yoav Wald, Gal Yona, Uri Shalit, and Yair Carmon. 2022. Malign overfitting: Interpolation and invariance are fundamentally at odds. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications. [find](https://scholar.google.com/scholar?q=Yoav+Wald%2C+Gal+Yona%2C+Uri+Shalit%2C+and+Yair+Carmon%2E+2022%2E+Malign+overfitting%3A+Interpolation+and+invariance+are+fundamentally+at+odds%2E+In+NeurIPS+2022+Workshop+on+Distribution+Shifts%3A+Connecting+Methods+)
57. Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 35413– 35425. [find](https://scholar.google.com/scholar?q=Alexander+Wan%2C+Eric+Wallace%2C+Sheng+Shen%2C+and+Dan+Klein%2E+2023%2E+Poisoning+language+models+during+instruction+tuning%2E+In+ICML%2C+volume+202+of+Proceedings+of+Machine+Learning+Research%2C+pages+35413%E2%80%93+35425%2E)
58. Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162. [find](https://scholar.google.com/scholar?q=Boyi+Wei%2C+Kaixuan+Huang%2C+Yangsibo+Huang%2C+Tinghao+Xie%2C+Xiangyu+Qi%2C+Mengzhou+Xia%2C+Prateek+Mittal%2C+Mengdi+Wang%2C+and+Peter+Henderson%2E+2024%2E+Assessing+the+brittleness+of+safety+alignment+via+pruning+and+lo)
59. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In ICLR.
60. Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more CRINGE than others: Preference optimization with the pairwise cringe loss. CoRR, abs/2312.16682. — [arXiv:2312.16682](https://arxiv.org/abs/2312.16682) [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Andrew+Lee%2C+Sainbayar+Sukhbaatar%2C+and+Jason+Weston%2E+2023%2E+Some+things+are+more+CRINGE+than+others%3A+Preference+optimization+with+the+pairwise+cringe+loss%2E+CoRR%2C+abs%2F2312%2E16682%2E+%E2%80%94+%5BarXiv%3A2312%2E1)

*Updated 31 October 2026*