---
title: "Language Models are Homer Simpson! Safety Re-Alignment of"
description: "The authors propose RESTA (REstoring Safety through Task Arithmetic), which repairs this damage… 49 references, cited by 45 papers here."
authors: ["Rishabh Bhardwaj", "Duc Anh Do", "Soujanya Poria"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.762/"
doi: "10.18653/v1/2024.acl-long.762"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bhardwaj-2024-language-models-are-homer-simpson/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bhardwaj-2024-language-models-are-homer-simpson/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic

# Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic

**Authors:** Rishabh Bhardwaj, Duc Anh Do and [Soujanya Poria](https://md.research.elamur.ai/free/en/authors/soujanya-poria/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.762/) · [PDF](https://aclanthology.org/2024.acl-long.762.pdf) · [DOI 10.18653/v1/2024.acl-long.762](https://doi.org/10.18653/v1/2024.acl-long.762) · [OpenAlex](https://openalex.org/W4402671137)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [45](https://md.research.elamur.ai/free/en/papers/bhardwaj-2024-language-models-are-homer-simpson/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Fine-tuning a safety-aligned LLM on ordinary, harmless data tends to weaken its safety guardrails. The authors propose RESTA (REstoring Safety through Task Arithmetic), which repairs this damage after fine-tuning. They compute a safety vector as the weight difference between a safety-aligned base model and its unaligned counterpart. They then add this vector, scaled by a factor γ, to the parameters of the fine-tuned model.

As an optional first step, DARE randomly zeroes the fine-tuning delta parameters and rescales the rest. In many settings, especially PEFT, this makes the correction stronger.

Experiments use Llama-2-7B, tuned with both PEFT and full fine-tuning. The training data covers instruction following in Chinese, English and Hindi, plus Code and Math. GPT-4 judges safety on three existing benchmarks and on CatQA, a new multilingual collection of harmful questions.

RESTA substantially reduces harmfulness while keeping most task performance. The gains extend to harm categories and languages outside the original setup. Because no additional training is needed, it is a cheap realignment step for anyone fine-tuning open models.

## Key points
- Averaged over HarmfulQA, AdversarialQA and DangerousQA and all fine-tuning domains, harmfulness fell from 18.59% to 5.14% (PEFT) and from 9.16% to 1.55% (Full-FT).
- CatQA has 550 harmful questions: 11 prohibited-use categories drawn from OpenAI and Meta policies, each with 5 sub-categories of 10 questions. It is available in English, Chinese and Vietnamese.
- On English CatQA, the average unsafety score dropped from 33.57% to 12.17% (PEFT) and from 22.16% to 4.34% (Full-FT).
- In other languages, the reductions were 26.2% (PEFT) and 21.37% (Full-FT) on Vietnamese CatQA, and 17.35% and 24.54% on Chinese CatQA.
- The authors list three limitations: no tests on larger models such as Llama-2-70B, limited tuning of the hyperparameters γ and p, and no study of transferring safety vectors between models. Code is released at github.com/declare-lab/resta.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Singapore University of Technology and Design](https://md.research.elamur.ai/free/en/universities/singapore-university-of-technology-and-design/) — University, Singapore, Singapore · 49 works · 11,074 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works · [Ancient Greek](https://md.research.elamur.ai/free/en/topics/ancient-greek/) · 14 works

## Cite

Cite as: Rishabh Bhardwaj, Duc Anh Do and Soujanya Poria. 2024. Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bhardwaj2024language,
  title = {{Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic}},
  author = {Rishabh Bhardwaj, Duc Anh Do and Soujanya Poria},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.762},
  url = {https://doi.org/10.18653/v1/2024.acl-long.762}
}
```

- [← previous Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://md.research.elamur.ai/free/en/papers/cheng-2024-instruction-pre-training-language-models/)
- [next → MAPO: Advancing Multilingual Reasoning through Multilingual-Alignment-as-Preference Optimization](https://md.research.elamur.ai/free/en/papers/she-2024-mapo-advancing-multilingual-reasoning-through/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

49 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
2. Rishabh Bhardwaj, Yingting Li, Navonil Majumder, Bo Cheng, and Soujanya Poria. 2023a. knn-cm: A non-parametric inference-phase adaptation of parametric text classifiers. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13546–13557. [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj%2C+Yingting+Li%2C+Navonil+Majumder%2C+Bo+Cheng%2C+and+Soujanya+Poria%2E+2023a%2E+knn%2Dcm%3A+A+non%2Dparametric+inference%2Dphase+adaptation+of+parametric+text+classifiers%2E+In+Findings+of+the+Association)
3. Rishabh Bhardwaj and Soujanya Poria. 2023a. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. arXiv preprint arXiv:2310.14303. — [arXiv:2310.14303](https://arxiv.org/abs/2310.14303) [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj+and+Soujanya+Poria%2E+2023a%2E+Language+model+unalignment%3A+Parametric+red%2Dteaming+to+expose+hidden+harms+and+biases%2E+arXiv+preprint+arXiv%3A2310%2E14303%2E+%E2%80%94+%5BarXiv%3A2310%2E14303%5D%28https%3A%2F%2Farxiv%2Eor)
4. Rishabh Bhardwaj and Soujanya Poria. 2023b. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662. [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj+and+Soujanya+Poria%2E+2023b%2E+Red%2Dteaming+large+language+models+using+chain+of+utterances+for+safety%2Dalignment%2E+arXiv+preprint+arXiv%3A2308%2E09662%2E)
5. Rishabh Bhardwaj, Tushar Vaidya, and Soujanya Poria. 2021. Knot: Knowledge distillation using optimal transport for solving nlp tasks. arXiv preprint arXiv:2110.02432. [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj%2C+Tushar+Vaidya%2C+and+Soujanya+Poria%2E+2021%2E+Knot%3A+Knowledge+distillation+using+optimal+transport+for+solving+nlp+tasks%2E+arXiv+preprint+arXiv%3A2110%2E02432%2E)
6. Rishabh Bhardwaj, Tushar Vaidya, and Soujanya Poria. 2022. Towards solving nlp tasks with optimal transport loss. Journal of King Saud University - Computer and Information Sciences, 34(10, Part B):10434–10443. [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj%2C+Tushar+Vaidya%2C+and+Soujanya+Poria%2E+2022%2E+Towards+solving+nlp+tasks+with+optimal+transport+loss%2E+Journal+of+King+Saud+University+%2D+Computer+and+Information+Sciences%2C+34%2810%2C+Part+B%29%3A10)
7. Rishabh Bhardwaj, Tushar Vaidya, and Soujanya Poria. 2023b. Adapter pruning using tropical characterization. arXiv preprint arXiv:2310.19232. [find](https://scholar.google.com/scholar?q=Rishabh+Bhardwaj%2C+Tushar+Vaidya%2C+and+Soujanya+Poria%2E+2023b%2E+Adapter+pruning+using+tropical+characterization%2E+arXiv+preprint+arXiv%3A2310%2E19232%2E)
8. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. 2023. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Milad+Nasr%2C+Christopher+A+Choquette%2DChoo%2C+Matthew+Jagielski%2C+Irena+Gao%2C+Anas+Awadalla%2C+Pang+Wei+Koh%2C+Daphne+Ippolito%2C+Katherine+Lee%2C+Florian+Tramer%2C+et+al%2E+2023%2E+Are+aligned+neural+n)
9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
10. Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. 2022. Fusing finetuned models for better pretraining. https://arxiv.org/abs/2204. 03044. [↗](https://arxiv.org/abs/2204)
11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
12. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
13. Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. 2020. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Advances in Neural Information Processing… [find](https://scholar.google.com/scholar?q=Stanislav+Fort%2C+Gintare+Karolina+Dziugaite%2C+Mansheej+Paul%2C+Sepideh+Kharaghani%2C+Daniel+M+Roy%2C+and+Surya+Ganguli%2E+2020%2E+Deep+learning+versus+kernel+learning%3A+an+empirical+study+of+loss+landscape+geometr)
14. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML). [https://proceedings.mlr.press/](https://proceedings.mlr.press/) v119/frankle20a.html. [↗](https://proceedings.mlr.press/)
15. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. [SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning](https://md.research.elamur.ai/free/en/papers/gordon-2012-semeval-2012-task-7-choice/). In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the s…
16. Patrick Haller, Ansar Aynetdinov, and Alan Akbik. 2023. Opiniongpt: Modelling explicit biases in instruction-tuned llms. arXiv preprint arXiv:2309.03876. [find](https://scholar.google.com/scholar?q=Patrick+Haller%2C+Ansar+Aynetdinov%2C+and+Alan+Akbik%2E+2023%2E+Opiniongpt%3A+Modelling+explicit+biases+in+instruction%2Dtuned+llms%2E+arXiv+preprint+arXiv%3A2309%2E03876%2E)
17. Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. 2023. Self-destructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 287–296. [find](https://scholar.google.com/scholar?q=Peter+Henderson%2C+Eric+Mitchell%2C+Christopher+Manning%2C+Dan+Jurafsky%2C+and+Chelsea+Finn%2E+2023%2E+Self%2Ddestructing+models%3A+Increasing+the+costs+of+harmful+dual+uses+of+foundation+models%2E+In+Proceedings+of+th)
18. Dominik Hintersdorf, Lukas Struppek, and Kristian Kersting. 2023. Balancing transparency and risk: The security and privacy risks of open-source machine learning models. arXiv preprint arXiv:2308.09490. — [arXiv:2308.09490](https://arxiv.org/abs/2308.09490) [find](https://scholar.google.com/scholar?q=Dominik+Hintersdorf%2C+Lukas+Struppek%2C+and+Kristian+Kersting%2E+2023%2E+Balancing+transparency+and+risk%3A+The+security+and+privacy+risks+of+open%2Dsource+machine+learning+models%2E+arXiv+preprint+arXiv%3A2308%2E0949)
19. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+arXiv+preprint+arXiv%3A1801%2E06146%2E)
20. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
21. Yiming Cui, Ziqing Yang, and Xin Yao. 2023. [Efficient and effective text encoding for chinese llama and alpaca](https://md.research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/). arXiv preprint arXiv:2304.08177.
22. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022a. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). arXiv preprint arXiv:2212.04089.
23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
24. Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022b. Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems (NeurIPS). [https://arXiv.org/](https://arXiv.org/) abs/2208.05… — [arXiv:2208.05592](https://arxiv.org/abs/2208.05592) [↗](https://arXiv.org/)
25. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2022. [On second thought, let’s not think step by step!](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/) bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061.
26. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. [Averaging weights leads to wider optima and better generalization](https://md.research.elamur.ai/free/en/papers/izmailov-2018-averaging-weights-leads-wider-optima/). In Conference on Uncertainty in Artificial Intelligence (UAI). [https://arxiv.org/](https://arxiv.org/) abs/1803.05407. [↗](https://arxiv.org/)
27. Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. 2023. On the exploitability of instruction tuning. arXiv preprint arXiv:2306.17194. — [arXiv:2306.17194](https://arxiv.org/abs/2306.17194) [find](https://scholar.google.com/scholar?q=Manli+Shu%2C+Jiongxiao+Wang%2C+Chen+Zhu%2C+Jonas+Geiping%2C+Chaowei+Xiao%2C+and+Tom+Goldstein%2E+2023%2E+On+the+exploitability+of+instruction+tuning%2E+arXiv+preprint+arXiv%3A2306%2E17194%2E+%E2%80%94+%5BarXiv%3A2306%2E17194%5D%28https%3A%2F%2Far)
28. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). arXiv preprint arXiv:2104.08691.
29. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
30. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models. https://arxiv.org/abs/2208.03306. [↗](https://arxiv.org/abs/2208.03306)
31. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023. Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124. — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) [find](https://scholar.google.com/scholar?q=Yuhui+Li%2C+Fangyun+Wei%2C+Jinjing+Zhao%2C+Chao+Zhang%2C+and+Hongyang+Zhang%2E+2023%2E+Rain%3A+Your+language+models+can+align+themselves+without+finetuning%2E+arXiv+preprint+arXiv%3A2309%2E07124%2E+%E2%80%94+%5BarXiv%3A2309%2E07124%5D%28htt)
32. Michael Matena and Colin Raffel. 2021. [Merging models with fisher-weighted averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/). In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2111. 09832. [↗](https://arxiv.org/abs/2111)
33. Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. 2020. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems (NeurIPS). [https://arxiv.org/](https://arxiv.org/) abs/2008.11687. — [arXiv:2008.11687](https://arxiv.org/abs/2008.11687) [↗](https://arxiv.org/)
34. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [XCOPA: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association…
35. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. [Fine-tuning aligned language models compromises safety, even when users do not intend to!](https://md.research.elamur.ai/free/en/papers/qi-2023-fine-tuning-aligned-language-models/) arXiv preprint arXiv:2310.03693.
36. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+OpenAI%2E)
37. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290.
38. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
39. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
40. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+pre)
41. Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. 2023. [Fundamental limitations of alignment in large language models](https://md.research.elamur.ai/free/en/papers/wolf-2023-fundamental-limitations-alignment-large-language/). arXiv preprint arXiv:2304.11082.
42. Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022a. [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://md.research.elamur.ai/free/en/papers/wortsman-2022-model-soups-averaging-weights-multiple/).…
43. Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. 2022b. Robust fine-tuning of zero-shot models. In Conference on Computer Vision and Pattern Recognition (CVPR). https://arxiv.org/abs/2109.01903. [↗](https://arxiv.org/abs/2109.01903)
44. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949. [find](https://scholar.google.com/scholar?q=Xianjun+Yang%2C+Xiao+Wang%2C+Qi+Zhang%2C+Linda+Petzold%2C+William+Yang+Wang%2C+Xun+Zhao%2C+and+Dahua+Lin%2E+2023%2E+Shadow+alignment%3A+The+ease+of+subverting+safely%2Daligned+language+models%2E+arXiv+preprint+arXiv%3A2310%2E0)
45. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023. [Language models are super mario: Absorbing abilities from homologous models as a free lunch](https://md.research.elamur.ai/free/en/papers/yu-2023-language-models-super-mario-absorbing/). arXiv preprint arXiv:2311.03099.
46. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. [Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). arXiv preprint arXiv:2106.10199.
47. Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023. [Llama-adapter: Efficient fine-tuning of language models with zero-init attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/). arXiv preprint arXiv:2303.16199.
48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685.
49. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/). arXiv preprint arXiv:2307.15043.

*Updated 30 October 2026*