---
title: "Machine Unlearning of Pre-trained Large Language Models (2024)"
description: "The paper addresses how to make a pre-trained large language model behave as if certain training… 76 references, cited by 49 papers here."
authors: ["Jin Yao", "Eli Chien", "Minxin Du", "Xinyao Niu", "Tianhao Wang", "Zezhou Cheng", "Xiang Yue"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.457/"
doi: "10.18653/v1/2024.acl-long.457"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yao-2024-machine-unlearning-pre-trained-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yao-2024-machine-unlearning-pre-trained-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Machine Unlearning of Pre-trained Large Language Models

# Machine Unlearning of Pre-trained Large Language Models

**Authors:** Jin Yao, Eli Chien, Minxin Du, [Xinyao Niu](https://md.research.elamur.ai/free/en/authors/xinyao-niu/), Tianhao Wang, Zezhou Cheng and [Xiang Yue](https://md.research.elamur.ai/free/en/authors/xiang-yue/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.457/) · [PDF](https://aclanthology.org/2024.acl-long.457.pdf) · [DOI 10.18653/v1/2024.acl-long.457](https://doi.org/10.18653/v1/2024.acl-long.457) · [OpenAlex](https://openalex.org/W4402671611)

**Citations:** 35 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/yao-2024-machine-unlearning-pre-trained-large/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses how to make a pre-trained large language model behave as if certain training documents had never been included, for example in response to copyright or privacy removal requests. Earlier unlearning work mostly targeted fine-tuned models, where retraining on the remaining data is still affordable. Retraining an entire pre-trained LLM is not affordable. The authors cast several first-order, gradient-based techniques as special cases of one shared objective and adapt seven of them to generative LLMs. These include gradient ascent, fine-tuning with random labels, adversarial-sample unlearning, and gradient ascent paired with descent or a KL term on retained data. Because exact retraining cannot serve as a reference, they compare against an "approximate retraining" baseline: the model's behaviour on unseen data drawn from the same distribution. Experiments with Yi-6B use forget sets from arXiv, books, and GitHub, consisting of thousands of 4096-token chunks. The methods run more than five orders of magnitude cheaper than retraining. The approach least affected by hyperparameter choices combined gradient ascent with descent steps on data from the same distribution as the forget set. This makes unlearning of pre-trained models a more realistic option.

## Key points
- Unlearning is reported as over 10^5 times more computationally efficient than retraining from scratch. Pre-training corpora reach about 10^12 tokens, compared with roughly 10^6 or fewer for fine-tuning sets.
- The benchmark covers three domains (arXiv, books, GitHub code) and uses thousands of 4096-token chunks. Earlier studies unlearned at most 128 samples.
- All experiments use Yi-6B. The released dataset includes genuine pre-training samples supplied by the Yi developers, and the code and data are public at github.com/yaojin17/Unlearning_LLM.
- Gradient ascent on the forget set combined with gradient descent on in-distribution retain data gave the most robust results across hyperparameter settings.
- Tuning advice: run a coarse learning-rate search with a step of 1e-5 between 5e-6 and 5e-5, then refine within the narrowed interval. Perplexity on the forget set rises as the learning rate increases.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 69 works · 22,788 citations
- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 60 works · 21,199 citations
- [Hong Kong Polytechnic University](https://md.research.elamur.ai/free/en/universities/hong-kong-polytechnic-university/) — University, Hong Kong, China · 34 works · 4,476 citations
- [University of Virginia](https://md.research.elamur.ai/free/en/universities/university-of-virginia/) — University, Charlottesville, United States · 18 works · 8,786 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng and Xiang Yue. 2024. Machine Unlearning of Pre-trained Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yao2024machine,
  title = {{Machine Unlearning of Pre-trained Large Language Models}},
  author = {Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng and Xiang Yue},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.457},
  url = {https://doi.org/10.18653/v1/2024.acl-long.457}
}
```

- [← previous Evaluating Mathematical Reasoning Beyond Accuracy](https://md.research.elamur.ai/free/en/papers/xia-2024-evaluating-mathematical-reasoning-accuracy/)
- [next → Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning](https://md.research.elamur.ai/free/en/papers/tan-2024-democratizing-large-language-models-personalized/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

76 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
2. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In S&P, pages 141–159. [find](https://scholar.google.com/scholar?q=Lucas+Bourtoule%2C+Varun+Chandrasekaran%2C+Christopher+A%2E+Choquette%2DChoo%2C+Hengrui+Jia%2C+Adelin+Travers%2C+Baiwu+Zhang%2C+David+Lie%2C+and+Nicolas+Papernot%2E+2021%2E+Machine+unlearning%2E+In+S%26P%2C+pages+141%E2%80%93159%2E)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). NeurIPS, 33:1877–1901.
4. Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In S&P, pages 463–480. [find](https://scholar.google.com/scholar?q=Yinzhi+Cao+and+Junfeng+Yang%2E+2015%2E+Towards+making+systems+forget+with+machine+unlearning%2E+In+S%26P%2C+pages+463%E2%80%93480%2E)
5. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. 2022. Membership inference attacks from first principles. In S&P, pages 1897–1914. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Steve+Chien%2C+Milad+Nasr%2C+Shuang+Song%2C+Andreas+Terzis%2C+and+Florian+Tram%C3%A8r%2E+2022%2E+Membership+inference+attacks+from+first+principles%2E+In+S%26P%2C+pages+1897%E2%80%931914%2E)
6. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In ICLR. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Florian+Tram%C3%A8r%2C+and+Chiyuan+Zhang%2E+2023%2E+Quantifying+memorization+across+neural+language+models%2E+In+ICLR%2E)
7. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. [The secret sharer: Evaluating and testing unintended memorization in neural networks](https://md.research.elamur.ai/free/en/papers/carlini-2018-secret-sharer-evaluating-testing-unintended/). In USENIX Security, pages 267–284.
8. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In USENIX Security, pages 2633–2650.
9. Sungmin Cha, Sungjun Cho, Dasol Hwang, Honglak Lee, Taesup Moon, and Moontae Lee. 2023. Learning to unlearn: Instance-wise unlearning for pre-trained classifiers. abs/2301.11578. — [arXiv:2301.11578](https://arxiv.org/abs/2301.11578) [find](https://scholar.google.com/scholar?q=Sungmin+Cha%2C+Sungjun+Cho%2C+Dasol+Hwang%2C+Honglak+Lee%2C+Taesup+Moon%2C+and+Moontae+Lee%2E+2023%2E+Learning+to+unlearn%3A+Instance%2Dwise+unlearning+for+pre%2Dtrained+classifiers%2E+abs%2F2301%2E11578%2E+%E2%80%94+%5BarXiv%3A2301%2E11578%5D%28)
10. Jiaao Chen and Diyi Yang. 2023. [Unlearn what you want to forget: Efficient unlearning for llms](https://md.research.elamur.ai/free/en/papers/chen-2023-unlearn-what-you-want-forget/). arXiv:2310.20150. EMNLP.
11. Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. [find](https://scholar.google.com/scholar?q=Matei+Zaharia%2C+and+Reynold+Xin%2E+2023%2E+Free+dolly%3A+Introducing+the+world%E2%80%99s+first+truly+open+instruction%2Dtuned+llm%2E)
12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
13. Minxin Du, Xiang Yue, Sherman SM Chow, and Huan Sun. 2023a. Sanitizing sentence embeddings (and labels) for local differential privacy. In TheWebConf, pages 2349–2359. [find](https://scholar.google.com/scholar?q=Minxin+Du%2C+Xiang+Yue%2C+Sherman+SM+Chow%2C+and+Huan+Sun%2E+2023a%2E+Sanitizing+sentence+embeddings+%28and+labels%29+for+local+differential+privacy%2E+In+TheWebConf%2C+pages+2349%E2%80%932359%2E)
14. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E+Gonzalez%2C+et+al%2E+2023%2E+Vicuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+wi)
15. Eli Chien, Chao Pan, and Olgica Milenkovic. 2023. Efficient model updates for approximate unlearning of graph-structured data. In ICLR. [find](https://scholar.google.com/scholar?q=Eli+Chien%2C+Chao+Pan%2C+and+Olgica+Milenkovic%2E+2023%2E+Efficient+model+updates+for+approximate+unlearning+of+graph%2Dstructured+data%2E+In+ICLR%2E)
16. Rishav Chourasia and Neil Shah. 2023. Forget unlearning: Towards true data-deletion in machine learning. In ICML, pages 6028–6073. [find](https://scholar.google.com/scholar?q=Rishav+Chourasia+and+Neil+Shah%2E+2023%2E+Forget+unlearning%3A+Towards+true+data%2Ddeletion+in+machine+learning%2E+In+ICML%2C+pages+6028%E2%80%936073%2E)
17. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In NeurIPS, pages 4299–4307. [find](https://scholar.google.com/scholar?q=Paul+F%2E+Christiano%2C+Jan+Leike%2C+Tom+B%2E+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+NeurIPS%2C+pages+4299%E2%80%934307%2E)
18. Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan S. Kankanhalli. 2023. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In AAAI, pages 7210–7217. [find](https://scholar.google.com/scholar?q=Vikram+S%2E+Chundawat%2C+Ayush+K%2E+Tarun%2C+Murari+Mandal%2C+and+Mohan+S%2E+Kankanhalli%2E+2023%2E+Can+bad+teaching+induce+forgetting%3F+unlearning+in+deep+networks+using+an+incompetent+teacher%2E+In+AAAI%2C+pages+7210%E2%80%9372)
19. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arX)
20. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
21. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. 2023b. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In CCS, pag… [find](https://scholar.google.com/scholar?q=Mike+Conover%2C+Matt+Hayes%2C+Ankit+Mathur%2C+Jianwei+Xie%2C+Jun+Wan%2C+Sam+Shah%2C+Ali+Ghodsi%2C+Patrick+Wendell%2C+Minxin+Du%2C+Xiang+Yue%2C+Sherman+SM+Chow%2C+Tianhao+Wang%2C+Chenyu+Huang%2C+and+Huan+Sun%2E+2023b%2E+Dp%2Dforward%3A)
22. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. [Do membership inference attacks work on large language models?](https://md.research.elamur.ai/free/en/papers/duan-2024-membership-inference-attacks-work-large/)
23. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. 2006. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265–284. [find](https://scholar.google.com/scholar?q=Cynthia+Dwork%2C+Frank+McSherry%2C+Kobbi+Nissim%2C+and+Adam+D%2E+Smith%2E+2006%2E+Calibrating+noise+to+sensitivity+in+private+data+analysis%2E+In+TCC%2C+pages+265%E2%80%93284%2E)
24. Cynthia Dwork and Aaron Roth. 2014. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211–407. [find](https://scholar.google.com/scholar?q=Cynthia+Dwork+and+Aaron+Roth%2E+2014%2E+The+algorithmic+foundations+of+differential+privacy%2E+Found%2E+Trends+Theor%2E+Comput%2E+Sci%2E%2C+9%283%2D4%29%3A211%E2%80%93407%2E)
25. Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms. arXiv:2310.02238. [find](https://scholar.google.com/scholar?q=Ronen+Eldan+and+Mark+Russinovich%2E+2023%2E+Who%E2%80%99s+harry+potter%3F+approximate+unlearning+in+llms%2E+arXiv%3A2310%2E02238%2E)
26. Sanjam Garg, Shafi Goldwasser, and Prashant Nalini Vasudevan. 2020. Formalizing data deletion in the context of the right to be forgotten. In EUROCRYPT, pages 373–402. [find](https://scholar.google.com/scholar?q=Sanjam+Garg%2C+Shafi+Goldwasser%2C+and+Prashant+Nalini+Vasudevan%2E+2020%2E+Formalizing+data+deletion+in+the+context+of+the+right+to+be+forgotten%2E+In+EUROCRYPT%2C+pages+373%E2%80%93402%2E)
27. Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. 2019. Making AI forget you: Data deletion in machine learning. In NeurIPS, pages 3513– 3526. [find](https://scholar.google.com/scholar?q=Antonio+Ginart%2C+Melody+Y%2E+Guan%2C+Gregory+Valiant%2C+and+James+Zou%2E+2019%2E+Making+AI+forget+you%3A+Data+deletion+in+machine+learning%2E+In+NeurIPS%2C+pages+3513%E2%80%93+3526%2E)
28. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In CVPR, pages 9301–9309. [find](https://scholar.google.com/scholar?q=Aditya+Golatkar%2C+Alessandro+Achille%2C+and+Stefano+Soatto%2E+2020%2E+Eternal+sunshine+of+the+spotless+net%3A+Selective+forgetting+in+deep+networks%2E+In+CVPR%2C+pages+9301%E2%80%939309%2E)
29. Chen Gong, Kecen Li, Jin Yao, and Tianhao Wang. 2024. Trajdeleter: Enabling trajectory forgetting in offline reinforcement learning agents. arXiv preprint arXiv:2404.12530. — [arXiv:2404.12530](https://arxiv.org/abs/2404.12530) [find](https://scholar.google.com/scholar?q=Chen+Gong%2C+Kecen+Li%2C+Jin+Yao%2C+and+Tianhao+Wang%2E+2024%2E+Trajdeleter%3A+Enabling+trajectory+forgetting+in+offline+reinforcement+learning+agents%2E+arXiv+preprint+arXiv%3A2404%2E12530%2E+%E2%80%94+%5BarXiv%3A2404%2E12530%5D%28https%3A)
30. Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. 2020. Certified data removal from machine learning models. In ICML, pages 3832– 3842. [find](https://scholar.google.com/scholar?q=Chuan+Guo%2C+Tom+Goldstein%2C+Awni+Y%2E+Hannun%2C+and+Laurens+van+der+Maaten%2E+2020%2E+Certified+data+removal+from+machine+learning+models%2E+In+ICML%2C+pages+3832%E2%80%93+3842%2E)
31. Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Sharifi-Malvajerdi, and Chris Waites. 2021. Adaptive machine unlearning. In NeurIPS, pages 16319–16330. [find](https://scholar.google.com/scholar?q=Varun+Gupta%2C+Christopher+Jung%2C+Seth+Neel%2C+Aaron+Roth%2C+Saeed+Sharifi%2DMalvajerdi%2C+and+Chris+Waites%2E+2021%2E+Adaptive+machine+unlearning%2E+In+NeurIPS%2C+pages+16319%E2%80%9316330%2E)
32. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In ICLR.
33. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In ICML, pages 2790–2799. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
34. Kecen Li, Chen Gong, Zhixiang Li, Yuzhong Zhao, Xinwen Hou, and Tianhao Wang. 2024. Meticulously selecting 1% of the dataset for pre-training! generating differentially private images data with semantics query. USENIX Security. [find](https://scholar.google.com/scholar?q=Kecen+Li%2C+Chen+Gong%2C+Zhixiang+Li%2C+Yuzhong+Zhao%2C+Xinwen+Hou%2C+and+Tianhao+Wang%2E+2024%2E+Meticulously+selecting+1%25+of+the+dataset+for+pre%2Dtraining%21+generating+differentially+private+images+data+with+semant)
35. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In ICLR.
36. Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679. — [arXiv:2110.05679](https://arxiv.org/abs/2110.05679) [find](https://scholar.google.com/scholar?q=Xuechen+Li%2C+Florian+Tramer%2C+Percy+Liang%2C+and+Tatsunori+Hashimoto%2E+2021%2E+Large+language+models+can+be+strong+differentially+private+learners%2E+arXiv+preprint+arXiv%3A2110%2E05679%2E+%E2%80%94+%5BarXiv%3A2110%2E05679%5D%28https)
37. Yiyang Huang and Clément L. Canonne. 2023. Tight bounds for machine unlearning via differential privacy. arXiv:2309.00886. — [arXiv:2309.00886](https://arxiv.org/abs/2309.00886) [find](https://scholar.google.com/scholar?q=Yiyang+Huang+and+Cl%C3%A9ment+L%2E+Canonne%2E+2023%2E+Tight+bounds+for+machine+unlearning+via+differential+privacy%2E+arXiv%3A2309%2E00886%2E+%E2%80%94+%5BarXiv%3A2309%2E00886%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E00886%29)
38. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
39. Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. 2021. Approximate data deletion from machine learning models. In AISTATS, pages 2008–2016. [find](https://scholar.google.com/scholar?q=Zachary+Izzo%2C+Mary+Anne+Smart%2C+Kamalika+Chaudhuri%2C+and+James+Zou%2E+2021%2E+Approximate+data+deletion+from+machine+learning+models%2E+In+AISTATS%2C+pages+2008%E2%80%932016%2E)
40. Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, and Chiyuan Zhang. 2023. Measuring forgetting of memorized training examples. In ICLR. [find](https://scholar.google.com/scholar?q=Matthew+Jagielski%2C+Om+Thakkar%2C+Florian+Tram%C3%A8r%2C+Daphne+Ippolito%2C+Katherine+Lee%2C+Nicholas+Carlini%2C+Eric+Wallace%2C+Shuang+Song%2C+Abhradeep+Guha+Thakurta%2C+Nicolas+Papernot%2C+and+Chiyuan+Zhang%2E+2023%2E+Measurin)
41. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. [Knowledge unlearning for mitigating privacy risks in language models](https://md.research.elamur.ai/free/en/papers/jang-2022-knowledge-unlearning-mitigating-privacy-risks/). In ACL, pages 14389–14408.
42. Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. 2023. Model sparsification can simplify machine unlearning. In NeurIPS (Spotlight). ArXiv:2304.04934. — [arXiv:2304.04934](https://arxiv.org/abs/2304.04934) [find](https://scholar.google.com/scholar?q=Jinghan+Jia%2C+Jiancheng+Liu%2C+Parikshit+Ram%2C+Yuguang+Yao%2C+Gaowen+Liu%2C+Yang+Liu%2C+Pranay+Sharma%2C+and+Sijia+Liu%2E+2023%2E+Model+sparsification+can+simplify+machine+unlearning%2E+In+NeurIPS+%28Spotlight%29%2E+ArXiv%3A23)
43. Ronald Kemker, Marc McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. 2018. Measuring catastrophic forgetting in neural networks. In AAAI, pages 3390–3398. [find](https://scholar.google.com/scholar?q=Ronald+Kemker%2C+Marc+McClure%2C+Angelina+Abitino%2C+Tyler+L%2E+Hayes%2C+and+Christopher+Kanan%2E+2018%2E+Measuring+catastrophic+forgetting+in+neural+networks%2E+In+AAAI%2C+pages+3390%E2%80%933398%2E)
44. Hyunjik Kim, George Papamakarios, and Andriy Mnih. 2021. The lipschitz constant of self-attention. In ICML, pages 5562–5571. [find](https://scholar.google.com/scholar?q=Hyunjik+Kim%2C+George+Papamakarios%2C+and+Andriy+Mnih%2E+2021%2E+The+lipschitz+constant+of+self%2Dattention%2E+In+ICML%2C+pages+5562%E2%80%935571%2E)
45. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023.… — [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
46. Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. 2022. Privacy adhering machine un-learning in NLP. arXiv:2212.09573. [find](https://scholar.google.com/scholar?q=Vinayshekhar+Bannihatti+Kumar%2C+Rashmi+Gangadharaiah%2C+and+Dan+Roth%2E+2022%2E+Privacy+adhering+machine+un%2Dlearning+in+NLP%2E+arXiv%3A2212%2E09573%2E)
47. Meghdad Kurmanji, Peter Triantafillou, and Eleni Triantafillou. 2023. Towards unbounded machine unlearning. arXiv:2302.09880. — [arXiv:2302.09880](https://arxiv.org/abs/2302.09880) [find](https://scholar.google.com/scholar?q=Meghdad+Kurmanji%2C+Peter+Triantafillou%2C+and+Eleni+Triantafillou%2E+2023%2E+Towards+unbounded+machine+unlearning%2E+arXiv%3A2302%2E09880%2E+%E2%80%94+%5BarXiv%3A2302%2E09880%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302%2E09880%29)
48. Zheyuan Liu, Guangyao Dou, Yijun Tian, Chunhui Zhang, Eli Chien, and Ziwei Zhu. 2024. Breaking the trilemma of privacy, utility, efficiency via controllable machine unlearning. [find](https://scholar.google.com/scholar?q=Zheyuan+Liu%2C+Guangyao+Dou%2C+Yijun+Tian%2C+Chunhui+Zhang%2C+Eli+Chien%2C+and+Ziwei+Zhu%2E+2024%2E+Breaking+the+trilemma+of+privacy%2C+utility%2C+efficiency+via+controllable+machine+unlearning%2E)
49. Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2022. Estimating the carbon footprint of bloom, a 176b parameter language model. arXiv:2211.02001. — [arXiv:2211.02001](https://arxiv.org/abs/2211.02001) [find](https://scholar.google.com/scholar?q=Alexandra+Sasha+Luccioni%2C+Sylvain+Viguier%2C+and+Anne%2DLaure+Ligozat%2E+2022%2E+Estimating+the+carbon+footprint+of+bloom%2C+a+176b+parameter+language+model%2E+arXiv%3A2211%2E02001%2E+%E2%80%94+%5BarXiv%3A2211%2E02001%5D%28https%3A%2F%2Farxiv)
50. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. [Tofu: A task of fictitious unlearning for llms](https://md.research.elamur.ai/free/en/papers/maini-2024-tofu-task-fictitious-unlearning-llms/). arXiv preprint arXiv:2401.06121.
51. Yunlong Mao, Zexi Xin, Zhenyu Li, Jue Hong, Qingyou Yang, and Sheng Zhong. 2023. Secure split learning against property inference, data reconstruction, and feature space hijacking attacks. arXiv preprint arXiv:2304.09515. — [arXiv:2304.09515](https://arxiv.org/abs/2304.09515) [find](https://scholar.google.com/scholar?q=Yunlong+Mao%2C+Zexi+Xin%2C+Zhenyu+Li%2C+Jue+Hong%2C+Qingyou+Yang%2C+and+Sheng+Zhong%2E+2023%2E+Secure+split+learning+against+property+inference%2C+data+reconstruction%2C+and+feature+space+hijacking+attacks%2E+arXiv+prepr)
52. Ilya Mironov. 2017. Rényi differential privacy. In CSF, pages 263–275. [find](https://scholar.google.com/scholar?q=Ilya+Mironov%2E+2017%2E+R%C3%A9nyi+differential+privacy%2E+In+CSF%2C+pages+263%E2%80%93275%2E)
53. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
54. Alexandra Peste, Dan Alistarh, and Christoph H. Lampert. 2021. SSSE: efficiently erasing samples from trained machine learning models. arXiv:2107.03860. — [arXiv:2107.03860](https://arxiv.org/abs/2107.03860) [find](https://scholar.google.com/scholar?q=Alexandra+Peste%2C+Dan+Alistarh%2C+and+Christoph+H%2E+Lampert%2E+2021%2E+SSSE%3A+efficiently+erasing+samples+from+trained+machine+learning+models%2E+arXiv%3A2107%2E03860%2E+%E2%80%94+%5BarXiv%3A2107%2E03860%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2107)
55. RealTimeData. 2024. github_latest. [find](https://scholar.google.com/scholar?q=RealTimeData%2E+2024%2E+github%5Flatest%2E)
56. Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what you want to forget: Algorithms for machine unlearning. In NeurIPS, pages 18075–18086. [find](https://scholar.google.com/scholar?q=Ayush+Sekhari%2C+Jayadev+Acharya%2C+Gautam+Kamath%2C+and+Ananda+Theertha+Suresh%2E+2021%2E+Remember+what+you+want+to+forget%3A+Algorithms+for+machine+unlearning%2E+In+NeurIPS%2C+pages+18075%E2%80%9318086%2E)
57. Chenze Shao and Yang Feng. 2022. Overcoming catastrophic forgetting beyond continual learning: Balanced training for neural machine translation. In ACL, pages 2023–2036. [find](https://scholar.google.com/scholar?q=Chenze+Shao+and+Yang+Feng%2E+2022%2E+Overcoming+catastrophic+forgetting+beyond+continual+learning%3A+Balanced+training+for+neural+machine+translation%2E+In+ACL%2C+pages+2023%E2%80%932036%2E)
58. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Anirudh+Ajith%2C+Mengzhou+Xia%2C+Yangsibo+Huang%2C+Daogao+Liu%2C+Terra+Blevins%2C+Danqi+Chen%2C+and+Luke+Zettlemoyer%2E+2023%2E+Detecting+pretraining+data+from+large+language+models%2E+arXiv+preprint+arXiv%3A)
59. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. [Membership inference attacks against machine learning models](https://md.research.elamur.ai/free/en/papers/shokri-2016-membership-inference-attacks-against-machine/). In S&P, pages 3–18.
60. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. In NeurIPS. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F%2E+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+In+NeurIPS%2E)
61. Ayush K. Tarun, Vikram S. Chundawat, Murari Mandal, and Mohan S. Kankanhalli. 2021. Fast yet effective machine unlearning. CoRR, abs/2111.08947. — [arXiv:2111.08947](https://arxiv.org/abs/2111.08947) [find](https://scholar.google.com/scholar?q=Ayush+K%2E+Tarun%2C+Vikram+S%2E+Chundawat%2C+Murari+Mandal%2C+and+Mohan+S%2E+Kankanhalli%2E+2021%2E+Fast+yet+effective+machine+unlearning%2E+CoRR%2C+abs%2F2111%2E08947%2E+%E2%80%94+%5BarXiv%3A2111%2E08947%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2111%2E08947%29)
62. Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. [Memorization without overfitting: Analyzing the training dynamics of large language models](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/). In NeurIPS.
63. TogetherComputer. 2023. Redpajama: An open source recipe to reproduce llama training dataset. [find](https://scholar.google.com/scholar?q=TogetherComputer%2E+2023%2E+Redpajama%3A+An+open+source+recipe+to+reproduce+llama+training+dataset%2E)
64. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv:2…
65. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton%2DFe)
66. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/).
67. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. [Bloomberggpt: A large language model for finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/). arXiv preprint arXiv:2303.17564.
68. Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S. Yu. 2024. Machine unlearning: A survey. ACM Comput. Surv., 56(1):9:1–9:36. [find](https://scholar.google.com/scholar?q=Heng+Xu%2C+Tianqing+Zhu%2C+Lefeng+Zhang%2C+Wanlei+Zhou%2C+and+Philip+S%2E+Yu%2E+2024%2E+Machine+unlearning%3A+A+survey%2E+ACM+Comput%2E+Surv%2E%2C+56%281%29%3A9%3A1%E2%80%939%3A36%2E)
69. Borui Yang, Wei Li, Liyao Xiang, and Bo Li. 2023. Towards code watermarking with dual-channel transformations. arXiv preprint arXiv:2309.00860. — [arXiv:2309.00860](https://arxiv.org/abs/2309.00860) [find](https://scholar.google.com/scholar?q=Borui+Yang%2C+Wei+Li%2C+Liyao+Xiang%2C+and+Bo+Li%2E+2023%2E+Towards+code+watermarking+with+dual%2Dchannel+transformations%2E+arXiv+preprint+arXiv%3A2309%2E00860%2E+%E2%80%94+%5BarXiv%3A2309%2E00860%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E00860%29)
70. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. [Large language model unlearning](https://md.research.elamur.ai/free/en/papers/yao-2023-large-language-model-unlearning/). arXiv preprint arXiv:2310.10683.
71. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. [Yi: Open foundation models by 01](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/). ai. arXiv preprint arXiv:2403.04652.
72. Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, and Sherman SM Chow. 2021. Differential privacy for text analytics via natural text sanitization. arXiv preprint arXiv:2106.01221. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Minxin+Du%2C+Tianhao+Wang%2C+Yaliang+Li%2C+Huan+Sun%2C+and+Sherman+SM+Chow%2E+2021%2E+Differential+privacy+for+text+analytics+via+natural+text+sanitization%2E+arXiv+preprint+arXiv%3A2106%2E01221%2E)
73. Xiang Yue, Huseyin A Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim. 2022. Synthetic text generation with differential privacy: A simple and practical recipe. arXiv preprint arXiv:2210.14348. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Huseyin+A+Inan%2C+Xuechen+Li%2C+Girish+Kumar%2C+Julia+McAnallen%2C+Hoda+Shajari%2C+Huan+Sun%2C+David+Levitan%2C+and+Robert+Sim%2E+2022%2E+Synthetic+text+generation+with+differential+privacy%3A+A+simple+and+pra)
74. Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. 2023a. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. arXiv:2307.03941. — [arXiv:2307.03941](https://arxiv.org/abs/2307.03941) [find](https://scholar.google.com/scholar?q=Dawen+Zhang%2C+Pamela+Finckenberg%2DBroman%2C+Thong+Hoang%2C+Shidong+Pan%2C+Zhenchang+Xing%2C+Mark+Staples%2C+and+Xiwei+Xu%2E+2023a%2E+Right+to+be+forgotten+in+the+era+of+large+language+models%3A+Implications%2C+challenges)
75. Xulong Zhang, Jianzong Wang, Ning Cheng, Yifu Sun, Chuanyao Zhang, and Jing Xiao. 2023b. Machine unlearning methodology base on stochastic teacher network. arXiv:2308.14322. — [arXiv:2308.14322](https://arxiv.org/abs/2308.14322) [find](https://scholar.google.com/scholar?q=Xulong+Zhang%2C+Jianzong+Wang%2C+Ning+Cheng%2C+Yifu+Sun%2C+Chuanyao+Zhang%2C+and+Jing+Xiao%2E+2023b%2E+Machine+unlearning+methodology+base+on+stochastic+teacher+network%2E+arXiv%3A2308%2E14322%2E+%E2%80%94+%5BarXiv%3A2308%2E14322%5D%28https)
76. Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. 2024. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. arXiv preprint arXiv:2402.05813. — [arXiv:2402.05813](https://arxiv.org/abs/2402.05813) [find](https://scholar.google.com/scholar?q=Lingzhi+Wang%2C+Xingshan+Zeng%2C+Jinsong+Guo%2C+Kam%2DFai+Wong%2C+and+Georg+Gottlob%2E+2024%2E+Selective+forgetting%3A+Advancing+machine+unlearning+techniques+and+evaluation+in+language+models%2E+arXiv+preprint+arXiv%3A2)

*Updated 24 October 2026*