---
title: "Towards Safer Large Language Models through Machine"
description: "The authors propose Selective Knowledge negation Unlearning (SKU), a two-stage framework that… 38 references, cited by 49 papers here."
authors: ["Zheyuan Liu", "Guangyao Dou", "Zhaoxuan Tan", "Yijun Tian", "Meng Jiang"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: ACL 2024"
source: "https://aclanthology.org/2024.findings-acl.107/"
doi: "10.18653/v1/2024.findings-acl.107"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-safer-large-language-models-through/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-safer-large-language-models-through/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Towards Safer Large Language Models through Machine Unlearning

# Towards Safer Large Language Models through Machine Unlearning

**Authors:** [Zheyuan Liu](https://md.research.elamur.ai/free/en/authors/zheyuan-liu/), Guangyao Dou, [Zhaoxuan Tan](https://md.research.elamur.ai/free/en/authors/zhaoxuan-tan/), [Yijun Tian](https://md.research.elamur.ai/free/en/authors/yijun-tian/) and [Meng Jiang](https://md.research.elamur.ai/free/en/authors/meng-jiang/) · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-acl.107/) · [PDF](https://aclanthology.org/2024.findings-acl.107.pdf) · [DOI 10.18653/v1/2024.findings-acl.107](https://doi.org/10.18653/v1/2024.findings-acl.107) · [OpenAlex](https://openalex.org/W4402670860)

**Citations:** 22 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/liu-2024-safer-large-language-models-through/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large language models can still give dangerous answers to harmful prompts. Earlier unlearning methods based on gradient ascent suppress such answers but tend to damage replies to ordinary requests. The authors propose Selective Knowledge negation Unlearning (SKU), a two-stage framework that removes harmful knowledge while keeping the model useful.

In the first stage, a copy of the pretrained model is deliberately pushed toward harmfulness by three modules:
- **guided distortion**: cross-entropy training on harmful prompt–response pairs;
- **random disassociation**: training on shuffled, mismatched harmful pairs to capture more varied harmful content;
- **preservation divergence**: a negative KL term on normal prompts.

The weight difference between this "bad" model and the original forms a task vector. In the second stage, that vector is subtracted from the pretrained weights, following the task-vector negation idea of Ilharco et al.

Across several LLM architectures (detailed here for LLAMA2-7B), SKU does not win any single metric. It does give the best overall balance between a low harmful rate and preserved response quality. This makes it a cheaper safety option than RLHF.

## Key points
- On LLAMA2-7B, SKU lowers the harmful rate on unlearned samples from 44% to 3%. GA with Mismatch reaches a similar drop (47% → 3.5%), but its perplexity exceeds 10³ by 1000 steps, so its outputs become incoherent.
- SKU's perplexity rises only from 23.92 to 24.86 during training (the pretrained LLAMA2-7B scores 19.84).
- SKU's BLEURT score falls only from -1.10 to -1.211, versus -1.324 to -1.995 for GA + Mismatch.
- Plain fine-tuning (FT) keeps good perplexity but barely changes harmfulness (57% → 53%). Naive task-vector negation only gets from 54% to 35%, showing that simple fine-tune-then-negate is insufficient.
- Stated limitations: robustness to jailbreak-style adversarial attacks and to general Right To Be Forgotten settings is untested. Code is released at github.com/franciscoliu/SKU.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 170 works · 60,739 citations
- [University of Notre Dame](https://md.research.elamur.ai/free/en/universities/university-of-notre-dame/) — University, Notre Dame, United States · 24 works · 36,415 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 209 works

## Cite

Cite as: Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian and Meng Jiang. 2024. Towards Safer Large Language Models through Machine Unlearning. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2024safer,
  title = {{Towards Safer Large Language Models through Machine Unlearning}},
  author = {Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian and Meng Jiang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.107},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.107}
}
```

- [← previous LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://md.research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/)
- [next → Aligning Large and Small Language Models via Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/ranaldi-2024-aligning-large-small-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

38 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2022. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836. — [arXiv:2209.04836](https://arxiv.org/abs/2209.04836) [find](https://scholar.google.com/scholar?q=Samuel+K+Ainsworth%2C+Jonathan+Hayase%2C+and+Siddhartha+Srinivasa%2E+2022%2E+Git+re%2Dbasin%3A+Merging+models+modulo+permutation+symmetries%2E+arXiv+preprint+arXiv%3A2209%2E04836%2E+%E2%80%94+%5BarXiv%3A2209%2E04836%5D%28https%3A%2F%2Farxiv%2Eorg)
3. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP). [find](https://scholar.google.com/scholar?q=Lucas+Bourtoule%2C+Varun+Chandrasekaran%2C+Christopher+A+Choquette%2DChoo%2C+Hengrui+Jia%2C+Adelin+Travers%2C+Baiwu+Zhang%2C+David+Lie%2C+and+Nicolas+Papernot%2E+2021%2E+Machine+unlearning%2E+In+2021+IEEE+Symposium+on+Secu)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Neurips.
5. Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy. [find](https://scholar.google.com/scholar?q=Yinzhi+Cao+and+Junfeng+Yang%2E+2015%2E+Towards+making+systems+forget+with+machine+unlearning%2E+In+2015+IEEE+symposium+on+security+and+privacy%2E)
6. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217. [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Xander+Davies%2C+Claudia+Shi%2C+Thomas+Krendl+Gilbert%2C+J%C3%A9r%C3%A9my+Scheurer%2C+Javier+Rando%2C+Rachel+Freedman%2C+Tomasz+Korbak%2C+David+Lindner%2C+Pedro+Freire%2C+et+al%2E+2023%2E+Open+problems+and+fundamenta)
7. Jiaao Chen and Diyi Yang. 2023. [Unlearn what you want to forget: Efficient unlearning for llms](https://md.research.elamur.ai/free/en/papers/chen-2023-unlearn-what-you-want-forget/). arXiv preprint arXiv:2310.20150.
8. Eli Chien, Chao Pan, and Olgica Milenkovic. 2022. Efficient model updates for approximate unlearning of graph-structured data. In ICLR. [find](https://scholar.google.com/scholar?q=Eli+Chien%2C+Chao+Pan%2C+and+Olgica+Milenkovic%2E+2022%2E+Efficient+model+updates+for+approximate+unlearning+of+graph%2Dstructured+data%2E+In+ICLR%2E)
9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research.
10. Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. 2023. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In AAAI. [find](https://scholar.google.com/scholar?q=Vikram+S+Chundawat%2C+Ayush+K+Tarun%2C+Murari+Mandal%2C+and+Mohan+Kankanhalli%2E+2023%2E+Can+bad+teaching+induce+forgetting%3F+unlearning+in+deep+networks+using+an+incompetent+teacher%2E+In+AAAI%2E)
11. Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238. [find](https://scholar.google.com/scholar?q=Ronen+Eldan+and+Mark+Russinovich%2E+2023%2E+Who%E2%80%99s+harry+potter%3F+approximate+unlearning+in+llms%2E+arXiv+preprint+arXiv%3A2310%2E02238%2E)
12. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In ICML. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle%2C+Gintare+Karolina+Dziugaite%2C+Daniel+Roy%2C+and+Michael+Carbin%2E+2020%2E+Linear+mode+connectivity+and+the+lottery+ticket+hypothesis%2E+In+ICML%2E)
13. Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems. [find](https://scholar.google.com/scholar?q=Antonio+Ginart%2C+Melody+Guan%2C+Gregory+Valiant%2C+and+James+Y+Zou%2E+2019%2E+Making+ai+forget+you%3A+Data+deletion+in+machine+learning%2E+Advances+in+neural+information+processing+systems%2E)
14. Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. 2020. Certified data removal from machine learning models. In ICML. [find](https://scholar.google.com/scholar?q=Chuan+Guo%2C+Tom+Goldstein%2C+Awni+Hannun%2C+and+Laurens+Van+Der+Maaten%2E+2020%2E+Certified+data+removal+from+machine+learning+models%2E+In+ICML%2E)
15. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
16. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022a. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). arXiv preprint arXiv:2212.04089.
17. Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022b. Patching open-vocabulary models by interpolating weights. Neurips. [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Samir+Yitzhak+Gadre%2C+Shuran+Song%2C+Hannaneh+Hajishirzi%2C+Simon+Kornblith%2C+Ali+Farhadi%2C+and+Ludwig+Schmidt%2E+2022b%2E+Patching+open%2Dvocabulary+models+by+interpolating+wei)
18. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. [Beavertails: Towards improved safety alignment of llm via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). arXiv preprint arXiv:2307.04657.
19. Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. 2023. Model sparsification can simplify machine unlearning. arXiv preprint arXiv:2304.04934. — [arXiv:2304.04934](https://arxiv.org/abs/2304.04934) [find](https://scholar.google.com/scholar?q=Jinghan+Jia%2C+Jiancheng+Liu%2C+Parikshit+Ram%2C+Yuguang+Yao%2C+Gaowen+Liu%2C+Yang+Liu%2C+Pranay+Sharma%2C+and+Sijia+Liu%2E+2023%2E+Model+sparsification+can+simplify+machine+unlearning%2E+arXiv+preprint+arXiv%3A2304%2E04934%2E)
20. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). Neurips.
21. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/), 2022. URL https://arxiv.org/abs/2206.14858. [↗](https://arxiv.org/abs/2206.14858)
22. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
23. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958.
24. Zheyuan Liu, Guangyao Dou, Yijun Tian, Chunhui Zhang, Eli Chien, and Ziwei Zhu. 2023. Breaking the trilemma of privacy, utility, efficiency via controllable machine unlearning. arXiv preprint arXiv:2310.18574. — [arXiv:2310.18574](https://arxiv.org/abs/2310.18574) [find](https://scholar.google.com/scholar?q=Zheyuan+Liu%2C+Guangyao+Dou%2C+Yijun+Tian%2C+Chunhui+Zhang%2C+Eli+Chien%2C+and+Ziwei+Zhu%2E+2023%2E+Breaking+the+trilemma+of+privacy%2C+utility%2C+efficiency+via+controllable+machine+unlearning%2E+arXiv+preprint+arXiv%3A23)
25. Zheyuan Liu, Xiaoxin He, Yijun Tian, and Nitesh V Chawla. 2024. Can we soft prompt llms for graph learning tasks? In WWW. [find](https://scholar.google.com/scholar?q=Zheyuan+Liu%2C+Xiaoxin+He%2C+Yijun+Tian%2C+and+Nitesh+V+Chawla%2E+2024%2E+Can+we+soft+prompt+llms+for+graph+learning+tasks%3F+In+WWW%2E)
26. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. [Tofu: A task of fictitious unlearning for llms](https://md.research.elamur.ai/free/en/papers/maini-2024-tofu-task-fictitious-unlearning-llms/). arXiv preprint arXiv:2401.06121.
27. Michael Matena and Colin Raffel. 2021. [Merging models with fisher-weighted averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/). arXiv preprint arXiv:2111.09832.
28. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) arXiv preprint arXiv:2202.12837.
29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Neurips.
30. Chao Pan, Eli Chien, and Olgica Milenkovic. 2023. Unlearning graph classifiers with limited data resources. In WWW. [find](https://scholar.google.com/scholar?q=Chao+Pan%2C+Eli+Chien%2C+and+Olgica+Milenkovic%2E+2023%2E+Unlearning+graph+classifiers+with+limited+data+resources%2E+In+WWW%2E)
31. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. [Is chatgpt a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) arXiv preprint arXiv:2302.06476.
32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2E)
33. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
34. Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what you want to forget: Algorithms for machine unlearning. Neurips. [find](https://scholar.google.com/scholar?q=Ayush+Sekhari%2C+Jayadev+Acharya%2C+Gautam+Kamath%2C+and+Ananda+Theertha+Suresh%2E+2021%2E+Remember+what+you+want+to+forget%3A+Algorithms+for+machine+unlearning%2E+Neurips%2E)
35. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+P+Parikh%2E+2020%2E+Bleurt%3A+Learning+robust+metrics+for+text+generation%2E+arXiv+preprint+arXiv%3A2004%2E04696%2E)
36. Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. 2024. [Democratizing large language models via personalized parameter-efficient fine-tuning](https://md.research.elamur.ai/free/en/papers/tan-2024-democratizing-large-language-models-personalized/). arXiv preprint arXiv:2402.04401.
37. Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. 2022. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). [find](https://scholar.google.com/scholar?q=Anvith+Thudi%2C+Gabriel+Deza%2C+Varun+Chandrasekaran%2C+and+Nicolas+Papernot%2E+2022%2E+Unrolling+sgd%3A+Understanding+factors+influencing+machine+unlearning%2E+In+2022+IEEE+7th+European+Symposium+on+Security+and+P)
38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)

*Updated 24 October 2026*