---
title: "Knowledge Sanitization of Large Language Models (2023)"
description: "Ishibashi and Shimodaira propose \"knowledge sanitization,\" which instead teaches the model to give… 29 references, cited by 17 papers here."
authors: ["Yoichi Ishibashi", "Hidetoshi Shimodaira"]
year: 2023
venue: "arXiv:2309.11852"
source: "https://arxiv.org/abs/2309.11852"
doi: "10.48550/arxiv.2309.11852"
arxiv: "2309.11852"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ishibashi-2023-knowledge-sanitization-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ishibashi-2023-knowledge-sanitization-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Knowledge Sanitization of Large Language Models

# Knowledge Sanitization of Large Language Models

**Authors:** Yoichi Ishibashi and Hidetoshi Shimodaira · **Year:** 2023 · **Venue:** arXiv:2309.11852 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.11852) · [PDF](https://arxiv.org/pdf/2309.11852) · [DOI 10.48550/arxiv.2309.11852](https://doi.org/10.48550/arxiv.2309.11852) · [arXiv 2309.11852](https://arxiv.org/abs/2309.11852) · [OpenAlex](https://openalex.org/W4386977575)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [17](https://md.research.elamur.ai/free/en/papers/ishibashi-2023-knowledge-sanitization-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models trained on web data can memorize private or confidential details and reveal them through extraction attacks or jailbreak prompts. Earlier unlearning methods suppress such facts, but the model may then produce a fabricated substitute, which is a hallucination. Ishibashi and Shimodaira propose "knowledge sanitization," which instead teaches the model to give a fixed safe reply, such as "I don't know," whenever a protected fact is requested. They fine-tune only the MLP weight matrices with LoRA, since prior work suggests factual knowledge is stored mainly in those layers. Training mixes questions whose answers are relabeled with the safe phrase and ordinary question–answer pairs that the model should keep answering correctly. Experiments use LLaMA and GPT-J on closed-book TriviaQA. The sanitized models refuse on the targeted facts while largely keeping their performance on other knowledge and on broader benchmarks. They also resist sentence-completion extraction attacks better than the original model and a ROME-edited model. The approach works on already-trained LLMs without retraining from scratch, offering a lightweight privacy safeguard.

## Key points
- **Method:** LoRA adapters are applied only to MLP layers. The training set combines a sanitized set K_S, in which answers to forget are replaced with "I don't know," and a retention set K_R, so the model learns when refusal is appropriate.
- **Data:** TriviaQA (about 95K QA pairs) is used in closed-book, zero-shot mode. Five answers are chosen at random as forgetting targets (e.g., "Rudyard Kipling"), with 16 training questions each, giving 80 pairs in K_F.
- **Models:** LLaMA and GPT-J are evaluated on forgetting and retention, as well as on commonsense reasoning and reading-comprehension tasks.
- **Robustness test:** GPT-4-written prompts try to elicit Julius Caesar and related facts (Calpurnia, Cleopatra). The sanitized LLaMA withholds these, while unrelated facts such as Pompeii and Agrippina are still answered. ROME and the original model leak more.
- **Resource:** Code and data are published at github.com/yoichi1484/knowledge-sanitization.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Kyoto University](https://md.research.elamur.ai/free/en/universities/kyoto-university/) — University, Kyoto, Japan · 40 works · 8,235 citations
- [RIKEN Center for Advanced Intelligence Project](https://md.research.elamur.ai/free/en/labs/riken-center-for-advanced-intelligence-project/) — Research institute, Tokyo, Japan · 15 works · 957 citations
- [RIKEN](https://md.research.elamur.ai/free/en/labs/riken/) — Research institute, Wako, Japan · 26 works · 526 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yoichi Ishibashi and Hidetoshi Shimodaira. 2023. Knowledge Sanitization of Large Language Models. arXiv:2309.11852.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ishibashi2023knowledge,
  title = {{Knowledge Sanitization of Large Language Models}},
  author = {Yoichi Ishibashi and Hidetoshi Shimodaira},
  year = {2023},
  doi = {10.48550/arxiv.2309.11852},
  eprint = {2309.11852},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.11852}
}
```

- [← previous Efficiently Adapting Pretrained Language Models To New Languages](https://md.research.elamur.ai/free/en/papers/csaki-2023-efficiently-adapting-pretrained-language-models/)
- [next → Large-scale Language Model Rescoring on Long-form Data](https://md.research.elamur.ai/free/en/papers/chen-2023-large-scale-language-model-rescoring/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pages 308–318. ACM. [find](https://scholar.google.com/scholar?q=Mart%C3%ADn+Abadi%2C+Andy+Chu%2C+Ian+J%2E+Goodfellow%2C+H%2E+Brendan+McMahan%2C+Ilya+Mironov%2C+Kunal+Talwar%2C+and+Li+Zhang%2E+2016%2E+Deep+learning+with+differential+privacy%2E+In+Proceedings+of+the+2016+ACM+SIGSAC+Conference)
2. David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hell… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
3. Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. [GPT-NeoX-20B: An open-source autor…](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
4. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). CoRR, abs/1803.05457.
7. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493– 8502.… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+Baobao+Chang%2C+and+Furu+Wei%2E+2022%2E+Knowledge+neurons+in+pretrained+transformers%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computationa)
8. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Florian+Tram%C3%A8r%2C+and+Chiyuan+Zhang%2E+2023%2E+Quantifying+memorization+across+neural+language+models%2E+In+The+Eleventh+International+Conf)
9. Cynthia Dwork. 2008. Differential privacy: A survey of results. In Theory and Applications of Models of Computation, 5th International Conference, TAMC 2008, Xi’an, China, April 25-29, 2008. Proceedings, volume 4978 of Lecture Notes in Computer Science, pages 1–19. Springer. [find](https://scholar.google.com/scholar?q=Cynthia+Dwork%2E+2008%2E+Differential+privacy%3A+A+survey+of+results%2E+In+Theory+and+Applications+of+Models+of+Computation%2C+5th+International+Conference%2C+TAMC+2008%2C+Xi%E2%80%99an%2C+China%2C+April+25%2D29%2C+2008%2E+Proceedin)
10. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In 30th USENIX Security Symposium, USENIX Security…
11. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
12. Sanjam Garg, Shafi Goldwasser, and Prashant Nalini Vasudevan. 2020. Formalizing data deletion in the context of the right to be forgotten. In Advances in Cryptology - EUROCRYPT 2020 - 39th Annual International Conference on the Theory and Applications of Cryptographic Techniques, Zagreb, Croatia, Ma… [find](https://scholar.google.com/scholar?q=Sanjam+Garg%2C+Shafi+Goldwasser%2C+and+Prashant+Nalini+Vasudevan%2E+2020%2E+Formalizing+data+deletion+in+the+context+of+the+right+to+be+forgotten%2E+In+Advances+in+Cryptology+%2D+EUROCRYPT+2020+%2D+39th+Annual+Inte)
13. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+Parker+Schuh%2C+Kensen+Shi%2C+Sasha+Tsvyashc)
14. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9301–9309. Computer Vision Foundati… [find](https://scholar.google.com/scholar?q=Aditya+Golatkar%2C+Alessandro+Achille%2C+and+Stefano+Soatto%2E+2020%2E+Eternal+sunshine+of+the+spotless+net%3A+Selective+forgetting+in+deep+networks%2E+In+2020+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+R)
15. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In NeurIPS. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+GPT%2E+In+NeurIPS%2E)
16. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018,… [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+A+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+)
17. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
18. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
19. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2038–2047. Association for Computat… [find](https://scholar.google.com/scholar?q=Jie+Huang%2C+Hanyin+Shao%2C+and+Kevin+Chen%2DChuan+Chang%2E+2022%2E+Are+large+pre%2Dtrained+language+models+leaking+your+personal+information%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2)
20. CoRR, Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. [ToTTo: A controlled table-to-text generation dataset](https://md.research.elamur.ai/free/en/papers/parikh-2020-totto-controlled-table-text-generation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173–1186, Online. As…
21. Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. [Editing models with task arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/). CoRR, abs/2212.04089.
22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
23. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. [Knowledge unlearning for mitigating privacy risks in language models](https://md.research.elamur.ai/free/en/papers/jang-2022-knowledge-unlearning-mitigating-privacy-risks/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2…
24. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+an+adversarial+winograd+schema+challenge+at+scale%2E+Commun%2E+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
25. David Sánchez and Montserrat Batet. 2014. C-sanitized: a privacy model for document redaction and sanitization. CoRR, abs/1406.4285. — [arXiv:1406.4285](https://arxiv.org/abs/1406.4285) [find](https://scholar.google.com/scholar?q=David+S%C3%A1nchez+and+Montserrat+Batet%2E+2014%2E+C%2Dsanitized%3A+a+privacy+model+for+document+redaction+and+sanitization%2E+CoRR%2C+abs%2F1406%2E4285%2E+%E2%80%94+%5BarXiv%3A1406%2E4285%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1406%2E4285%29)
26. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho… — [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
27. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4…
28. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. [RACE: large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–79…
29. Haochen Li, Tong Mo, Hongcheng Fan, Jingkun Wang, Jiaxi Wang, Fuhao Zhang, and Weiping Li. 2022. Kipt: Knowledge-injected prompt tuning for event detection. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, Hugo… [find](https://scholar.google.com/scholar?q=Haochen+Li%2C+Tong+Mo%2C+Hongcheng+Fan%2C+Jingkun+Wang%2C+Jiaxi+Wang%2C+Fuhao+Zhang%2C+and+Weiping+Li%2E+2022%2E+Kipt%3A+Knowledge%2Dinjected+prompt+tuning+for+event+detection%2E+In+Proceedings+of+the+29th+International+Co)

*Updated 30 October 2026*