---
title: "Large Language Model Unlearning — Yao 2023"
description: "The paper treats unlearning as a cheap alignment tool for LLMs."
authors: ["Yuanshun Yao", "Xiaojun Xu", "Yang Liu"]
year: 2023
venue: "arXiv:2310.10683"
source: "https://arxiv.org/abs/2310.10683"
doi: "10.48550/arxiv.2310.10683"
arxiv: "2310.10683"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yao-2023-large-language-model-unlearning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yao-2023-large-language-model-unlearning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / Large Language Model Unlearning

# Large Language Model Unlearning

**Authors:** Yuanshun Yao, Xiaojun Xu, [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/) · **Year:** 2023 · **Venue:** arXiv:2310.10683 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.10683) · [PDF](https://arxiv.org/pdf/2310.10683) · [DOI 10.48550/arxiv.2310.10683](https://doi.org/10.48550/arxiv.2310.10683) · [arXiv 2310.10683](https://arxiv.org/abs/2310.10683) · [OpenAlex](https://openalex.org/W4415796002)

**Citations:** 45 citations (openalex-snapshot · 2026-10-14) · [120](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper treats unlearning as a cheap alignment tool for LLMs. The idea is to make a model drop unwanted behaviour using only negative examples, such as harmful replies found through red teaming or user reports, without the human-written positive answers that RLHF needs. The authors define the setting, goals and evaluation criteria. Given a forget set of bad prompt–output pairs, the unlearned model should move its outputs away from those responses while keeping its usefulness on ordinary prompts. The method builds on gradient ascent (GA) over the forget set. A variant, GA+Mismatch, also trains toward random outputs, and both use weighted loss terms. They test three scenarios: removing harmful responses, erasing copyrighted material and forgetting hallucinated facts. In an ablation on OPT-1.3B with PKU-SafeRLHF, unlearning reaches harmful rates comparable to or below full RLHF while using about 2% of its compute. This makes it attractive for teams with tight budgets whose main concern is preventing damaging outputs.

## Key points
- **Advantages claimed:** only negative samples are required, collection of those samples can be largely automated, and the compute cost is close to light finetuning. The method also fits well when the offending training data is already known.
- **Harmful rate on OPT-1.3B / PKU-SafeRLHF (seen prompts):**
  - original model: 47%
  - SFT: 34%
  - full RLHF: 4%
  - GA: 1%
  - GA+Mismatch: 6%
- **Harmful rate on unseen prompts:**
  - original model: 53%
  - full RLHF: 7.5%
  - GA: 3%
  - GA+Mismatch: 7%
- **Trade-offs between the GA variants:** plain GA tends to produce degenerate text (diversity 0.118, fluency rated not meaningful). GA+Mismatch stays fluent and scores the best utility reward on normal prompts (−2.982, compared with −3.212 for RLHF).
- **Templated outputs and code:** replacing random targets with a fixed refusal ("I can't assist it.") gives similar results, with a 1% harmful rate on seen prompts. The code is available at github.com/kevinyaobytedance/llm_unlearn.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 117 not yet.

- [Investigating Memorization in Language Models Trained via Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/macking-2026-investigating-memorization-language-models-trained/)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://md.research.elamur.ai/free/en/papers/maini-2024-tofu-task-fictitious-unlearning-llms/)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://md.research.elamur.ai/free/en/papers/zhang-2024-negative-preference-optimization-catastrophic-collapse/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yao2023large,
  title = {{Large Language Model Unlearning}},
  author = {Yao, Yuanshun and Xu, Xiaojun and Liu, Yang},
  year = {2023},
  doi = {10.48550/arxiv.2310.10683},
  eprint = {2310.10683},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.10683}
}
```

- [← previous Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://md.research.elamur.ai/free/en/papers/ma-2023-large-language-model-is-not/)
- [next → Large Language Models are Better Reasoners with Self-Verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Bai%2C+S%2E+Kadavath%2C+S%2E+Kundu%2C+A%2E+Askell%2C+J%2E+Kernion%2C+A%2E+Jones%2C+A%2E+Chen%2C+A%2E+Goldie%2C+A%2E+Mirhoseini%2C+C%2E+McKinnon%2C+et+al%2E+Constitutional+ai%3A+Harmlessness+from+ai+feedback%2E+arXiv+preprint+arXiv%3A2212%2E08073) [our card](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
2. L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. [find](https://scholar.google.com/scholar?q=L%2E+Bourtoule%2C+V%2E+Chandrasekaran%2C+C%2E+A%2E+Choquette%2DChoo%2C+H%2E+Jia%2C+A%2E+Travers%2C+B%2E+Zhang%2C+D%2E+Lie%2C+and+N%2E+Papernot%2E+Machine+unlearning%2E+In+2021+IEEE+Symposium+on+Security+and+Privacy+%28SP%29%2C+pages+141%E2%80%93159%2E)
3. Y. Cao and J. Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pages 463–480. [find](https://scholar.google.com/scholar?q=Y%2E+Cao+and+J%2E+Yang%2E+Towards+making+systems+forget+with+machine+unlearning%2E+In+2015+IEEE+Symposium+on+Security+and+Privacy%2C+pages+463%E2%80%93480%2E)
4. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021. [find](https://scholar.google.com/scholar?q=N%2E+Carlini%2C+F%2E+Tramer%2C+E%2E+Wallace%2C+M%2E+Jagielski%2C+A%2E+Herbert%2DVoss%2C+K%2E+Lee%2C+A%2E+Roberts%2C+T%2E+B%2E+Brown%2C+D%2E+Song%2C+U%2E+Erlingsson%2C+et+al%2E+Extracting+training+data+from+large+language+models%2E+In+USENIX+Securit) [our card](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/)
5. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. [find](https://scholar.google.com/scholar?q=N%2E+Carlini%2C+D%2E+Ippolito%2C+M%2E+Jagielski%2C+K%2E+Lee%2C+F%2E+Tramer%2C+and+C%2E+Zhang%2E+Quantifying+memorization+across+neural+language+models%2E+arXiv+preprint+arXiv%3A2202%2E07646%2C+2022%2E)
6. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Christiano%2C+J%2E+Leike%2C+T%2E+Brown%2C+M%2E+Martic%2C+S%2E+Legg%2C+and+D%2E+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2C+2017%2E)
7. V. S. Chundawat, A. K. Tarun, M. Mandal, and M. Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 2023. [find](https://scholar.google.com/scholar?q=V%2E+S%2E+Chundawat%2C+A%2E+K%2E+Tarun%2C+M%2E+Mandal%2C+and+M%2E+Kankanhalli%2E+Zero%2Dshot+machine+unlearning%2E+IEEE+Transactions+on+Information+Forensics+and+Security%2C+2023%2E)
8. G. Copilot. Github copilot lawsuit. [https://www.courthousenews.com/microsoft-and-github-ask-court-to-scrap-lawsuit-over-ai-powered-copilot/](https://www.courthousenews.com/microsoft-and-github-ask-court-to-scrap-lawsuit-over-ai-powered-copilot/), 2023. [↗](https://www.courthousenews.com/microsoft-and-github-ask-court-to-scrap-lawsuit-over-ai-powered-copilot/)
9. R. Eldan and M. Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Eldan+and+M%2E+Russinovich%2E+Who%E2%80%99s+harry+potter%3F+approximate+unlearning+in+llms%2E+arXiv+preprint+arXiv%3A2310%2E02238%2C+2023%2E)
10. Facebook. Facebook community standards. [https://www.facebook.com/communitystandards/](https://www.facebook.com/communitystandards/), 2023. [↗](https://www.facebook.com/communitystandards/)
11. M. Grynbaum and R. Mac. The times sues openai and microsoft over a.i. use of copyrighted work. The New York Times, 2023. URL [https://www.nytimes.com/2023/12/27/business/media/](https://www.nytimes.com/2023/12/27/business/media/) new-york-times-open-ai-microsoft-lawsuit.html. Accessed: 2024-01-16. [↗](https://www.nytimes.com/2023/12/27/business/media/)
12. C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030, 2019. — [arXiv:1911.03030](https://arxiv.org/abs/1911.03030) [find](https://scholar.google.com/scholar?q=C%2E+Guo%2C+T%2E+Goldstein%2C+A%2E+Hannun%2C+and+L%2E+Van+Der+Maaten%2E+Certified+data+removal+from+machine+learning+models%2E+arXiv+preprint+arXiv%3A1911%2E03030%2C+2019%2E+%E2%80%94+%5BarXiv%3A1911%2E03030%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E0303)
13. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Holtzman%2C+J%2E+Buys%2C+L%2E+Du%2C+M%2E+Forbes%2C+and+Y%2E+Choi%2E+The+curious+case+of+neural+text+degeneration%2E+arXiv+preprint+arXiv%3A1904%2E09751%2C+2019%2E)
14. Z. Izzo, M. A. Smart, K. Chaudhuri, and J. Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008–2016. [find](https://scholar.google.com/scholar?q=Z%2E+Izzo%2C+M%2E+A%2E+Smart%2C+K%2E+Chaudhuri%2C+and+J%2E+Zou%2E+Approximate+data+deletion+from+machine+learning+models%2E+In+International+Conference+on+Artificial+Intelligence+and+Statistics%2C+pages+2008%E2%80%932016%2E)
15. J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, R. Sun, Y. Wang, and Y. Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. arXiv preprint arXiv:2307.04657, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Ji%2C+M%2E+Liu%2C+J%2E+Dai%2C+X%2E+Pan%2C+C%2E+Zhang%2C+C%2E+Bian%2C+R%2E+Sun%2C+Y%2E+Wang%2C+and+Y%2E+Yang%2E+Beavertails%3A+Towards+improved+safety+alignment+of+llm+via+a+human%2Dpreference+dataset%2E+arXiv+preprint+arXiv%3A2307%2E04657%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/)
16. J. Jia, J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. Sharma, and S. Liu. Model sparsification can simplify machine unlearning. arXiv preprint arXiv:2304.04934, 2023. — [arXiv:2304.04934](https://arxiv.org/abs/2304.04934) [find](https://scholar.google.com/scholar?q=J%2E+Jia%2C+J%2E+Liu%2C+P%2E+Ram%2C+Y%2E+Yao%2C+G%2E+Liu%2C+Y%2E+Liu%2C+P%2E+Sharma%2C+and+S%2E+Liu%2E+Model+sparsification+can+simplify+machine+unlearning%2E+arXiv+preprint+arXiv%3A2304%2E04934%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E04934%5D%28https%3A%2F%2Farxiv%2Eo)
17. J. Lee, T. Le, J. Chen, and D. Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pages 3637–3647, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Lee%2C+T%2E+Le%2C+J%2E+Chen%2C+and+D%2E+Lee%2E+Do+language+models+plagiarize%3F+In+Proceedings+of+the+ACM+Web+Conference+2023%2C+pages+3637%E2%80%933647%2C+2023%2E)
18. J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv e-prints, pages arXiv–2305, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Li%2C+X%2E+Cheng%2C+W%2E+X%2E+Zhao%2C+J%2E%2DY%2E+Nie%2C+and+J%2E%2DR%2E+Wen%2E+Halueval%3A+A+large%2Dscale+hallucination+evaluation+benchmark+for+large+language+models%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932305%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/)
19. S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. [find](https://scholar.google.com/scholar?q=S%2E+Lin%2C+J%2E+Hilton%2C+and+O%2E+Evans%2E+Truthfulqa%3A+Measuring+how+models+mimic+human+falsehoods%2E+arXiv+preprint+arXiv%3A2109%2E07958%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
20. Y. Liu, M. Fan, C. Chen, X. Liu, Z. Ma, L. Wang, and J. Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications, pages 280–289. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+M%2E+Fan%2C+C%2E+Chen%2C+X%2E+Liu%2C+Z%2E+Ma%2C+L%2E+Wang%2C+and+J%2E+Ma%2E+Backdoor+defense+with+machine+unlearning%2E+In+IEEE+INFOCOM+2022%2DIEEE+Conference+on+Computer+Communications%2C+pages+280%E2%80%93289%2E)
21. Y. Liu, Y. Yao, J.-F. Ton, X. Zhang, R. G. H. Cheng, Y. Klochkov, M. F. Taufiq, and H. Li. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+Y%2E+Yao%2C+J%2E%2DF%2E+Ton%2C+X%2E+Zhang%2C+R%2E+G%2E+H%2E+Cheng%2C+Y%2E+Klochkov%2C+M%2E+F%2E+Taufiq%2C+and+H%2E+Li%2E+Trustworthy+llms%3A+a+survey+and+guideline+for+evaluating+large+language+models%E2%80%99+alignment%2E+arXiv+preprint+arXi) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-trustworthy-llms-survey-guideline-evaluating/)
22. X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591–27609, 2022. [find](https://scholar.google.com/scholar?q=X%2E+Lu%2C+S%2E+Welleck%2C+J%2E+Hessel%2C+L%2E+Jiang%2C+L%2E+Qin%2C+P%2E+West%2C+P%2E+Ammanabrolu%2C+and+Y%2E+Choi%2E+Quark%3A+Controllable+text+generation+with+reinforced+unlearning%2E+Advances+in+neural+information+processing+systems%2C) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-quark-controllable-text-generation-reinforced/)
23. R. Müller, S. Kornblith, and G. E. Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=R%2E+M%C3%BCller%2C+S%2E+Kornblith%2C+and+G%2E+E%2E+Hinton%2E+When+does+label+smoothing+help%3F+Advances+in+neural+information+processing+systems%2C+32%2C+2019%2E)
24. K. P. Murphy. Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL probml.ai. [find](https://scholar.google.com/scholar?q=K%2E+P%2E+Murphy%2E+Probabilistic+Machine+Learning%3A+An+introduction%2E+MIT+Press%2C+2022%2E+URL+probml%2Eai%2E)
25. S. Neel, A. Roth, and S. Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pages 931–962. PMLR, 2021. [find](https://scholar.google.com/scholar?q=S%2E+Neel%2C+A%2E+Roth%2C+and+S%2E+Sharifi%2DMalvajerdi%2E+Descent%2Dto%2Ddelete%3A+Gradient%2Dbased+methods+for+machine+unlearning%2E+In+Algorithmic+Learning+Theory%2C+pages+931%E2%80%93962%2E+PMLR%2C+2021%2E)
26. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Ouyang%2C+J%2E+Wu%2C+X%2E+Jiang%2C+D%2E+Almeida%2C+C%2E+Wainwright%2C+P%2E+Mishkin%2C+C%2E+Zhang%2C+S%2E+Agarwal%2C+K%2E+Slama%2C+A%2E+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+Advances+in+Neura)
27. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E%2DJ%2E+Zhu%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+annual+meeting+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
28. M. Pawelczyk, S. Neel, and H. Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Pawelczyk%2C+S%2E+Neel%2C+and+H%2E+Lakkaraju%2E+In%2Dcontext+unlearning%3A+Language+models+as+few+shot+unlearners%2E+arXiv+preprint+arXiv%3A2310%2E07579%2C+2023%2E)
29. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+F%2E+Wolski%2C+P%2E+Dhariwal%2C+A%2E+Radford%2C+and+O%2E+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
30. T. Sellam, D. Das, and A. P. Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696, 2020. [find](https://scholar.google.com/scholar?q=T%2E+Sellam%2C+D%2E+Das%2C+and+A%2E+P%2E+Parikh%2E+Bleurt%3A+Learning+robust+metrics+for+text+generation%2E+arXiv+preprint+arXiv%3A2004%2E04696%2C+2020%2E)
31. R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. [find](https://scholar.google.com/scholar?q=R%2E+Shokri%2C+M%2E+Stronati%2C+C%2E+Song%2C+and+V%2E+Shmatikov%2E+Membership+inference+attacks+against+machine+learning+models%2E+In+2017+IEEE+symposium+on+security+and+privacy+%28SP%29%2C+pages+3%E2%80%9318%2E) [our card](https://md.research.elamur.ai/free/en/papers/shokri-2016-membership-inference-attacks-against-machine/)
32. Z. Small. Sarah silverman sues openai and meta over copyright infringement. The New York Times, 2023. URL [https://www.nytimes.com/2023/07/10/arts/](https://www.nytimes.com/2023/07/10/arts/) sarah-silverman-lawsuit-openai-meta.html. Accessed: 2024-01-16. [↗](https://www.nytimes.com/2023/07/10/arts/)
33. A. K. Tarun, V. S. Chundawat, M. Mandal, and M. Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023. [find](https://scholar.google.com/scholar?q=A%2E+K%2E+Tarun%2C+V%2E+S%2E+Chundawat%2C+M%2E+Mandal%2C+and+M%2E+Kankanhalli%2E+Fast+yet+effective+machine+unlearning%2E+IEEE+Transactions+on+Neural+Networks+and+Learning+Systems%2C+2023%2E)
34. A. Thudi, G. Deza, V. Chandrasekaran, and N. Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319. [find](https://scholar.google.com/scholar?q=A%2E+Thudi%2C+G%2E+Deza%2C+V%2E+Chandrasekaran%2C+and+N%2E+Papernot%2E+Unrolling+sgd%3A+Understanding+factors+influencing+machine+unlearning%2E+In+2022+IEEE+7th+European+Symposium+on+Security+and+Privacy+%28EuroS%26P%29%2C+pages)
35. TikTok. Tiktok community guidelines. [https://www.tiktok.com/community-guidelines?lang=](https://www.tiktok.com/community-guidelines?lang=) en, 2023. [↗](https://www.tiktok.com/community-guidelines?lang=)
36. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+preprint+arXiv%3A2)
37. Twitter. Twitter rules and policies. twitter-rules, 2023. [https://help.twitter.com/en/rules-and-policies/](https://help.twitter.com/en/rules-and-policies/) [↗](https://help.twitter.com/en/rules-and-policies/)
38. J. P. Wahle, T. Ruas, F. Kirstein, and B. Gipp. How large language models are transforming machine-paraphrased plagiarism. arXiv preprint arXiv:2210.03568, 2022. — [arXiv:2210.03568](https://arxiv.org/abs/2210.03568) [find](https://scholar.google.com/scholar?q=J%2E+P%2E+Wahle%2C+T%2E+Ruas%2C+F%2E+Kirstein%2C+and+B%2E+Gipp%2E+How+large+language+models+are+transforming+machine%2Dparaphrased+plagiarism%2E+arXiv+preprint+arXiv%3A2210%2E03568%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E03568%5D%28https%3A%2F%2Farxiv%2Eorg)
39. A. Warnecke, L. Pirch, C. Wressnegger, and K. Rieck. Machine unlearning of features and labels. arXiv preprint arXiv:2108.11577, 2021. — [arXiv:2108.11577](https://arxiv.org/abs/2108.11577) [find](https://scholar.google.com/scholar?q=A%2E+Warnecke%2C+L%2E+Pirch%2C+C%2E+Wressnegger%2C+and+K%2E+Rieck%2E+Machine+unlearning+of+features+and+labels%2E+arXiv+preprint+arXiv%3A2108%2E11577%2C+2021%2E+%E2%80%94+%5BarXiv%3A2108%2E11577%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2108%2E11577%29)
40. H. Xu, T. Zhu, L. Zhang, W. Zhou, and P. S. Yu. Machine unlearning: A survey. ACM Computing Surveys, 56(1):1–36, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Xu%2C+T%2E+Zhu%2C+L%2E+Zhang%2C+W%2E+Zhou%2C+and+P%2E+S%2E+Yu%2E+Machine+unlearning%3A+A+survey%2E+ACM+Computing+Surveys%2C+56%281%29%3A1%E2%80%9336%2C+2023%2E)
41. Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Yuan%2C+H%2E+Yuan%2C+C%2E+Tan%2C+W%2E+Wang%2C+S%2E+Huang%2C+and+F%2E+Huang%2E+Rrhf%3A+Rank+responses+to+align+language+models+with+human+feedback+without+tears%2E+arXiv+preprint+arXiv%3A2304%2E05302%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/)
42. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Zhang%2C+S%2E+Roller%2C+N%2E+Goyal%2C+M%2E+Artetxe%2C+M%2E+Chen%2C+S%2E+Chen%2C+C%2E+Dewan%2C+M%2E+Diab%2C+X%2E+Li%2C+X%2E+V%2E+Lin%2C+et+al%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models%2E+arXiv+preprint+arXiv%3A2205%2E01068%2C+2022%2E)
43. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. [find](https://scholar.google.com/scholar?q=T%2E+Zhang%2C+V%2E+Kishore%2C+F%2E+Wu%2C+K%2E+Q%2E+Weinberger%2C+and+Y%2E+Artzi%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2C+2019%2E)
44. R. Zheng, S. Dou, S. Gao, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, L. Xiong, L. Chen, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Zheng%2C+S%2E+Dou%2C+S%2E+Gao%2C+W%2E+Shen%2C+B%2E+Wang%2C+Y%2E+Liu%2C+S%2E+Jin%2C+Q%2E+Liu%2C+L%2E+Xiong%2C+L%2E+Chen%2C+et+al%2E+Secrets+of+rlhf+in+large+language+models+part+i%3A+Ppo%2E+arXiv+preprint+arXiv%3A2307%2E04964%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zheng-2023-secrets-rlhf-large-language-models/)
45. Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015. [find](https://scholar.google.com/scholar?q=Y%2E+Zhu%2C+R%2E+Kiros%2C+R%2E+Zemel%2C+R%2E+Salakhutdinov%2C+R%2E+Urtasun%2C+A%2E+Torralba%2C+and+S%2E+Fidler%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+and+reading+books%2E+In+The+IEE) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)
46. T. Y. Zhuo, Z. Li, Y. Huang, Y.-F. Li, W. Wang, G. Haffari, and F. Shiri. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. arXiv preprint arXiv:2301.12868, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Y%2E+Zhuo%2C+Z%2E+Li%2C+Y%2E+Huang%2C+Y%2E%2DF%2E+Li%2C+W%2E+Wang%2C+G%2E+Haffari%2C+and+F%2E+Shiri%2E+On+robustness+of+prompt%2Dbased+semantic+parsing+with+large+pre%2Dtrained+language+model%3A+An+empirical+study+on+codex%2E+arXiv+prepr)