---
title: "Analyzing Leakage of Personally Identifiable Information in"
description: "Lukas et al. define three levels of PII leakage through black-box API access as formal security games. 70 references, cited by 98 papers here."
authors: ["Nils Lukas", "Ahmed Salem", "Robert Sim", "Shruti Tople", "Lukas Wutschitz", "Santiago Zanella-Beguelin"]
year: 2023
venue: "arXiv:2302.00539"
source: "https://arxiv.org/abs/2302.00539"
doi: "10.48550/arxiv.2302.00539"
arxiv: "2302.00539"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lukas-2023-analyzing-leakage-personally-identifiable-information/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lukas-2023-analyzing-leakage-personally-identifiable-information/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Analyzing Leakage of Personally Identifiable Information in Language Models

# Analyzing Leakage of Personally Identifiable Information in Language Models

**Authors:** [Nils Lukas](https://md.research.elamur.ai/free/en/authors/nils-lukas/), [Ahmed Salem](https://md.research.elamur.ai/free/en/authors/ahmed-salem/), [Robert Sim](https://md.research.elamur.ai/free/en/authors/robert-sim/), Shruti Tople, Lukas Wutschitz and Santiago Zanella-Beguelin · **Year:** 2023 · **Venue:** [arXiv:2302.00539](https://md.research.elamur.ai/free/en/venues/proceedings-ieee-symposium-on-security-and-privacy-proceedings-of-the-ieee/) ([Proceedings - IEEE Symposium on Security and Privacy/Proceedings of the ... IEEE Symposium on Security and Privacy 2023](https://md.research.elamur.ai/free/en/venues/proceedings-ieee-symposium-on-security-and-privacy-proceedings-of-the-ieee/2023/)) · **Publisher:** [IEEE Computer Society](https://md.research.elamur.ai/free/en/publishers/ieee-computer-society/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2302.00539) · [PDF](https://arxiv.org/pdf/2302.00539) · [DOI 10.48550/arxiv.2302.00539](https://doi.org/10.48550/arxiv.2302.00539) · [arXiv 2302.00539](https://arxiv.org/abs/2302.00539) · [OpenAlex](https://openalex.org/W4385679821)

**Citations:** 154 citations (OpenAlex, 2026-10-14) · [98](https://md.research.elamur.ai/free/en/papers/lukas-2023-analyzing-leakage-personally-identifiable-information/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Fine-tuned language models can memorize and expose personal data. Many practitioners assume that NER-based scrubbing or differential privacy removes this risk, but that assumption has rarely been tested. Lukas et al. define three levels of PII leakage through black-box API access as formal security games. In extraction, the attacker knows nothing about the training data. In reconstruction, the attacker holds a sentence with the PII masked out. In inference, the attacker also has a list of candidate values.

The authors build an attack for each game and run them against GPT-2 models of several sizes. The models are fine-tuned on court cases (ECHR), corporate mail (Enron) and health-facility reviews (Yelp-Health), either undefended, scrubbed, trained with DP, or both. The reconstruction attack uses text on both sides of the mask, helped by a public masked LM, and recovers up to ten times more PII than earlier methods. Sentence-level DP sharply reduces leakage but still lets about 3% of PII sequences through. Scrubbing costs roughly as much utility as DP. The resulting metrics let practitioners tune the privacy/utility balance of a whole pipeline.

## Key points
- Datasets used for fine-tuning:
  - ECHR: about 118k records, 23.75% of them containing person names.
  - Enron: about 139k records, 81.45% with names.
  - Yelp-Health: about 79k records, 54.55% with names.
- PII is tagged with the Flair model `ner-english-ontonotes-large` across 21 entity classes, such as person, gpe, org, date, phone number and email address.
- Extraction success is measured with precision and recall over unique PII sequences. Reconstruction and inference are scored by top-1 accuracy on a randomly chosen masked PII.
- The extraction attacks achieve roughly twice the precision of prior work, even against DP-trained models. With DP (ε = 8), about 3% of PII sequences still leak.
- Prior work cited in the paper shows clinical NER recall can drop to 80% for some PII types (care unit numbers, versus 97% for names), so scrubbing leaves residual PII. Code is released at github.com/microsoft/analysing_pii_leakage.

## Interesting ideas
- **Tiered attacker knowledge.** The paper grades threats by how much the attacker knows: nothing, the surrounding context, or the context plus a candidate list. This gives a reusable template for privacy audits of any text model, including models trained on Ukrainian or other low-resource corpora.
- **Candidate ranking by model likelihood.** Each candidate is inserted into the masked slot and the target LM's probability of the full sentence is scored. This turns a generative model into a probe of what it has memorized, which is cheap to apply wherever only API access is available.
- **Filling extra masks left to right.** When a query contains several masked PII, a public masked LM fills them one at a time, keeping the sentence fluent before scoring. The trick is handy whenever off-the-shelf fill-mask tools cannot fill multiple slots jointly.
- **Evaluating defenses together.** Scrubbing and DP are compared alone and combined, with both leakage and perplexity reported side by side. This end-to-end view suggests designing a lighter scrubber that relies on DP for the rest, instead of over-scrubbing and losing utility.
- **Separate metrics for separate threats.** Extraction is scored with precision and recall, while reconstruction and inference use top-1 accuracy. Keeping these apart prevents one aggregate "memorization" number from hiding which attack actually succeeds.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 80 works · 15,030 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 626 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz and Santiago Zanella-Beguelin. 2023. Analyzing Leakage of Personally Identifiable Information in Language Models. arXiv:2302.00539.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lukas2023analyzing,
  title = {{Analyzing Leakage of Personally Identifiable Information in Language Models}},
  author = {Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz and Santiago Zanella-Beguelin},
  year = {2023},
  doi = {10.48550/arxiv.2302.00539},
  eprint = {2302.00539},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.00539}
}
```

- [← previous LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-leandojo-theorem-proving-retrieval-augmented/)
- [next → Personality Traits in Large Language Models](https://md.research.elamur.ai/free/en/papers/serapio-garcia-2023-personality-traits-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

70 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in 23rd ACM SIGSAC Conference on Computer and Communications Security, CCS. ACM, 2016, pp. 308–318. [find](https://scholar.google.com/scholar?q=M%2E+Abadi%2C+A%2E+Chu%2C+I%2E+Goodfellow%2C+H%2E+B%2E+McMahan%2C+I%2E+Mironov%2C+K%2E+Talwar%2C+and+L%2E+Zhang%2C+%E2%80%9CDeep+learning+with+differential+privacy%2C%E2%80%9D+in+23rd+ACM+SIGSAC+Conference+on+Computer+and+Communications+Security%2C+C)
2. A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, and R. Vollgraf, “[FLAIR: An easy-to-use framework for state-of-the-art NLP](https://md.research.elamur.ai/free/en/papers/akbik-2019-flair-easy-use-framework-state/),” in 2019 Conference of the North American Chapter of the ACL (demonstrations). ACL, 2019, pp. 54–59.
3. Apple, “Differential privacy,” Available online at [https://www.apple.com/privacy/docs/Differential](https://www.apple.com/privacy/docs/Differential) Privacy Overview.pdf, retrieved 28 Nov 2022. [↗](https://www.apple.com/privacy/docs/Differential)
4. S. Bird, E. Klein, and E. Loper, Natural language processing with Python: analyzing text with the Natural Language Toolkit. O’Reilly Media, 2009. [find](https://scholar.google.com/scholar?q=S%2E+Bird%2C+E%2E+Klein%2C+and+E%2E+Loper%2C+Natural+language+processing+with+Python%3A+analyzing+text+with+the+Natural+Language+Toolkit%2E+O%E2%80%99Reilly+Media%2C+2009%2E)
5. H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr, “What does it mean for a language model to preserve privacy?” in 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT. ACM, 2022, p. 2280–2292. [find](https://scholar.google.com/scholar?q=H%2E+Brown%2C+K%2E+Lee%2C+F%2E+Mireshghallah%2C+R%2E+Shokri%2C+and+F%2E+Trame%CC%80r%2C+%E2%80%9CWhat+does+it+mean+for+a+language+model+to+preserve+privacy%3F%E2%80%9D+in+2022+ACM+Conference+on+Fairness%2C+Accountability%2C+and+Transparency%2C+FAccT)
6. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “[Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/),” in Advances in neural information processing systems (NeurIPS), vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901.
7. P. Budzianowski and I. Vulić, “Hello, it’s GPT-2 – How can I help you? towards the use of pretrained language models for task-oriented dialogue systems,” arXiv preprint arXiv:1907.05774 [cs.CL], 2019. [find](https://scholar.google.com/scholar?q=P%2E+Budzianowski+and+I%2E+Vulic%CC%81%2C+%E2%80%9CHello%2C+it%E2%80%99s+GPT%2D2+%E2%80%93+How+can+I+help+you%3F+towards+the+use+of+pretrained+language+models+for+task%2Doriented+dialogue+systems%2C%E2%80%9D+arXiv+preprint+arXiv%3A1907%2E05774+%5Bcs%2ECL%5D%2C+2019)
8. N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song, “[The secret sharer: Evaluating and testing unintended memorization in neural networks](https://md.research.elamur.ai/free/en/papers/carlini-2018-secret-sharer-evaluating-testing-unintended/),” in 28th USENIX Security Symposium (USENIX Security 19), 2019, pp. 267– 284.
9. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “[Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/),” in 30th USENIX Security Symposium. USENIX Association, 2021, pp. 2633–2650.
10. N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, “[Membership inference attacks from first principles](https://md.research.elamur.ai/free/en/papers/carlini-2021-membership-inference-attacks-first-principles/),” in 43rd IEEE Symposium on Security and Privacy (S&P). IEEE, 2022, pp. 1897–1914.
11. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, “Quantifying memorization across neural language models,” arXiv preprint arXiv:2202.07646 [cs.CL], 2022. [find](https://scholar.google.com/scholar?q=N%2E+Carlini%2C+D%2E+Ippolito%2C+M%2E+Jagielski%2C+K%2E+Lee%2C+F%2E+Tramer%2C+and+C%2E+Zhang%2C+%E2%80%9CQuantifying+memorization+across+neural+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2202%2E07646+%5Bcs%2ECL%5D%2C+2022%2E)
12. I. Chalkidis, I. Androutsopoulos, and N. Aletras, “[Neural legal judgment prediction in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2019-neural-legal-judgment-prediction-english/),” in 57th Annual Meeting of the Association for Computational Linguistics, ACL. ACL, 2019, pp. 4317–4323.
13. M. X. Chen, B. N. Lee, G. Bansal, Y. Cao, S. Zhang, J. Lu, J. Tsay, Y. Wang, A. M. Dai, Z. Chen et al., “Gmail smart compose: Real-time assisted writing,” in 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 2287–2295. [find](https://scholar.google.com/scholar?q=M%2E+X%2E+Chen%2C+B%2E+N%2E+Lee%2C+G%2E+Bansal%2C+Y%2E+Cao%2C+S%2E+Zhang%2C+J%2E+Lu%2C+J%2E+Tsay%2C+Y%2E+Wang%2C+A%2E+M%2E+Dai%2C+Z%2E+Chen+et+al%2E%2C+%E2%80%9CGmail+smart+compose%3A+Real%2Dtime+assisted+writing%2C%E2%80%9D+in+25th+ACM+SIGKDD+International+Conference+o)
14. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “[BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/),” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT. ACL, 2019, pp. 4171–4186.
15. C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor, “Our data, ourselves: Privacy via distributed noise generation,” in Advances in Cryptology – EURO-CRYPT 2006. Springer, 2006, pp. 486–503. [find](https://scholar.google.com/scholar?q=C%2E+Dwork%2C+K%2E+Kenthapadi%2C+F%2E+McSherry%2C+I%2E+Mironov%2C+and+M%2E+Naor%2C+%E2%80%9COur+data%2C+ourselves%3A+Privacy+via+distributed+noise+generation%2C%E2%80%9D+in+Advances+in+Cryptology+%E2%80%93+EURO%2DCRYPT+2006%2E+Springer%2C+2006%2C+pp%2E+486%E2%80%93503)
16. C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of cryptography conference, TCC. Springer, 2006, pp. 265–284. [find](https://scholar.google.com/scholar?q=C%2E+Dwork%2C+F%2E+McSherry%2C+K%2E+Nissim%2C+and+A%2E+Smith%2C+%E2%80%9CCalibrating+noise+to+sensitivity+in+private+data+analysis%2C%E2%80%9D+in+Theory+of+cryptography+conference%2C+TCC%2E+Springer%2C+2006%2C+pp%2E+265%E2%80%93284%2E)
17. European Union, “Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation),”… [find](https://scholar.google.com/scholar?q=European+Union%2C+%E2%80%9CRegulation+%28EU%29+2016%2F679+of+the+European+Parliament+and+of+the+Council+of+27+april+2016+on+the+protection+of+natural+persons+with+regard+to+the+processing+of+personal+data+and+on+the+)
18. A. Fan, M. Lewis, and Y. Dauphin, “[Hierarchical neural story generation](https://md.research.elamur.ai/free/en/papers/fan-2018-hierarchical-neural-story-generation/),” in 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 2018, pp. 889–898.
19. W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022. [find](https://scholar.google.com/scholar?q=W%2E+Fedus%2C+B%2E+Zoph%2C+and+N%2E+Shazeer%2C+%E2%80%9CSwitch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2C%E2%80%9D+Journal+of+Machine+Learning+Research%2C+vol%2E+23%2C+no%2E+120%2C+pp%2E+1%E2%80%9339%2C+202)
20. V. Feldman, “Does learning require memorization? a short tale about a long tail,” in 52nd Annual ACM SIGACT Symposium on Theory of Computing. ACM, 2020, pp. 954–959. [find](https://scholar.google.com/scholar?q=V%2E+Feldman%2C+%E2%80%9CDoes+learning+require+memorization%3F+a+short+tale+about+a+long+tail%2C%E2%80%9D+in+52nd+Annual+ACM+SIGACT+Symposium+on+Theory+of+Computing%2E+ACM%2C+2020%2C+pp%2E+954%E2%80%93959%2E)
21. P. Golle, “Revisiting the uniqueness of simple demographics in the us population,” in 5th ACM Workshop on Privacy in Electronic Society. ACM, 2006, pp. 77–80. [find](https://scholar.google.com/scholar?q=P%2E+Golle%2C+%E2%80%9CRevisiting+the+uniqueness+of+simple+demographics+in+the+us+population%2C%E2%80%9D+in+5th+ACM+Workshop+on+Privacy+in+Electronic+Society%2E+ACM%2C+2006%2C+pp%2E+77%E2%80%9380%2E)
22. A. Hoang, A. Bosselut, A. Celikyilmaz, and Y. Choi, “Efficient adaptation of pretrained transformers for abstractive summarization,” arXiv preprint arXiv:1906.00138 [cs.CL], 2019. — [arXiv:1906.00138](https://arxiv.org/abs/1906.00138) [find](https://scholar.google.com/scholar?q=A%2E+Hoang%2C+A%2E+Bosselut%2C+A%2E+Celikyilmaz%2C+and+Y%2E+Choi%2C+%E2%80%9CEfficient+adaptation+of+pretrained+transformers+for+abstractive+summarization%2C%E2%80%9D+arXiv+preprint+arXiv%3A1906%2E00138+%5Bcs%2ECL%5D%2C+2019%2E+%E2%80%94+%5BarXiv%3A1906%2E00138%5D)
23. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-optim… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
24. M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, “spaCy: Industrial-strength natural language processing in Python,” 2020. [find](https://scholar.google.com/scholar?q=M%2E+Honnibal%2C+I%2E+Montani%2C+S%2E+Van+Landeghem%2C+and+A%2E+Boyd%2C+%E2%80%9CspaCy%3A+Industrial%2Dstrength+natural+language+processing+in+Python%2C%E2%80%9D+2020%2E)
25. E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, and R. Weischedel, “OntoNotes: The 90% solution,” in Human Language Technology Conference of the NAACL. ACL, 2006, pp. 57–60. [find](https://scholar.google.com/scholar?q=E%2E+Hovy%2C+M%2E+Marcus%2C+M%2E+Palmer%2C+L%2E+Ramshaw%2C+and+R%2E+Weischedel%2C+%E2%80%9COntoNotes%3A+The+90%25+solution%2C%E2%80%9D+in+Human+Language+Technology+Conference+of+the+NAACL%2E+ACL%2C+2006%2C+pp%2E+57%E2%80%9360%2E)
26. J. Huang, H. Shao, and K. C.-C. Chang, “Are large pre-trained language models leaking your personal information?” in Findings of the Association for Computational Linguistics, EMNLP. ACL, 2022, pp. 2038–2047. [find](https://scholar.google.com/scholar?q=J%2E+Huang%2C+H%2E+Shao%2C+and+K%2E+C%2E%2DC%2E+Chang%2C+%E2%80%9CAre+large+pre%2Dtrained+language+models+leaking+your+personal+information%3F%E2%80%9D+in+Findings+of+the+Association+for+Computational+Linguistics%2C+EMNLP%2E+ACL%2C+2022%2C+pp%2E+20)
27. T. Humphries, S. Oya, L. Tulloch, M. Rafuse, I. Goldberg, U. Hengartner, and F. Kerschbaum, “Investigating membership inference attacks under data dependencies,” arXiv preprint arXiv:2010.12112 [cs.CR], 2020. — [arXiv:2010.12112](https://arxiv.org/abs/2010.12112) [find](https://scholar.google.com/scholar?q=T%2E+Humphries%2C+S%2E+Oya%2C+L%2E+Tulloch%2C+M%2E+Rafuse%2C+I%2E+Goldberg%2C+U%2E+Hengartner%2C+and+F%2E+Kerschbaum%2C+%E2%80%9CInvestigating+membership+inference+attacks+under+data+dependencies%2C%E2%80%9D+arXiv+preprint+arXiv%3A2010%2E12112+%5Bcs%2ECR)
28. H. A. Inan, O. Ramadan, L. Wutschitz, D. Jones, V. Rühle, J. Withers, and R. Sim, “Training data leakage analysis in language models,” arXiv preprint arXiv:2101.05405v2 [cs.CR], 2021. — [arXiv:2101.05405](https://arxiv.org/abs/2101.05405) [find](https://scholar.google.com/scholar?q=H%2E+A%2E+Inan%2C+O%2E+Ramadan%2C+L%2E+Wutschitz%2C+D%2E+Jones%2C+V%2E+Ru%CC%88hle%2C+J%2E+Withers%2C+and+R%2E+Sim%2C+%E2%80%9CTraining+data+leakage+analysis+in+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2101%2E05405v2+%5Bcs%2ECR%5D%2C+2021%2E+%E2%80%94+%5BarXiv%3A2101%2E0)
29. D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini, “Preventing verbatim memorization in language models gives a false sense of privacy,” arXiv preprint arXiv:2210.17546 [cs.LG], 2022. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) [find](https://scholar.google.com/scholar?q=D%2E+Ippolito%2C+F%2E+Trame%CC%80r%2C+M%2E+Nasr%2C+C%2E+Zhang%2C+M%2E+Jagielski%2C+K%2E+Lee%2C+C%2E+A%2E+Choquette%2DChoo%2C+and+N%2E+Carlini%2C+%E2%80%9CPreventing+verbatim+memorization+in+language+models+gives+a+false+sense+of+privacy%2C%E2%80%9D+arXiv+prep)
30. A. Jagannatha, B. P. S. Rawat, and H. Yu, “Membership inference attack susceptibility of clinical language models,” arXiv preprint arXiv:2104.08305 [cs.CL], 2021. — [arXiv:2104.08305](https://arxiv.org/abs/2104.08305) [find](https://scholar.google.com/scholar?q=A%2E+Jagannatha%2C+B%2E+P%2E+S%2E+Rawat%2C+and+H%2E+Yu%2C+%E2%80%9CMembership+inference+attack+susceptibility+of+clinical+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2104%2E08305+%5Bcs%2ECL%5D%2C+2021%2E+%E2%80%94+%5BarXiv%3A2104%2E08305%5D%28https%3A%2F%2Farxiv%2Eor)
31. M. Jagielski, O. Thakkar, F. Tramèr, D. Ippolito, K. Lee, N. Carlini, E. Wallace, S. Song, A. Thakurta, N. Papernot et al., “Measuring forgetting of memorized training examples,” arXiv preprint arXiv:2207.00099 [cs.LG], 2022. — [arXiv:2207.00099](https://arxiv.org/abs/2207.00099) [find](https://scholar.google.com/scholar?q=M%2E+Jagielski%2C+O%2E+Thakkar%2C+F%2E+Trame%CC%80r%2C+D%2E+Ippolito%2C+K%2E+Lee%2C+N%2E+Carlini%2C+E%2E+Wallace%2C+S%2E+Song%2C+A%2E+Thakurta%2C+N%2E+Papernot+et+al%2E%2C+%E2%80%9CMeasuring+forgetting+of+memorized+training+examples%2C%E2%80%9D+arXiv+preprint+arXiv)
32. B. Jayaraman, E. Ghosh, M. Chase, S. Roy, H. Inan, W. Dai, and D. Evans, “Combing for credentials: Active pattern extraction from smart reply,” arXiv preprint arXiv:2203.07618 [cs.CR], 2022. — [arXiv:2203.07618](https://arxiv.org/abs/2203.07618) [find](https://scholar.google.com/scholar?q=B%2E+Jayaraman%2C+E%2E+Ghosh%2C+M%2E+Chase%2C+S%2E+Roy%2C+H%2E+Inan%2C+W%2E+Dai%2C+and+D%2E+Evans%2C+%E2%80%9CCombing+for+credentials%3A+Active+pattern+extraction+from+smart+reply%2C%E2%80%9D+arXiv+preprint+arXiv%3A2203%2E07618+%5Bcs%2ECR%5D%2C+2022%2E+%E2%80%94+%5BarXiv%3A)
33. N. Kandpal, E. Wallace, and C. Raffel, “[Deduplicating training data mitigates privacy risks in language models](https://md.research.elamur.ai/free/en/papers/kandpal-2022-deduplicating-training-data-mitigates-privacy/),” in 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 10 697–10 707.
34. B. Klimt and Y. Yang, “Introducing the Enron corpus,” in International Conference on Email and Anti-Spam, CEAS, vol. 45, 2004, pp. 92–96. [find](https://scholar.google.com/scholar?q=B%2E+Klimt+and+Y%2E+Yang%2C+%E2%80%9CIntroducing+the+Enron+corpus%2C%E2%80%9D+in+International+Conference+on+Email+and+Anti%2DSpam%2C+CEAS%2C+vol%2E+45%2C+2004%2C+pp%2E+92%E2%80%9396%2E)
35. G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “[Neural architectures for named entity recognition](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/),” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT. ACL, 2016, pp. 260–270.
36. J. Lee, T. Le, J. Chen, and D. Lee, “Do language models plagiarize?” arXiv preprint arXiv:2203.07618 [cs.CL], 2022. — [arXiv:2203.07618](https://arxiv.org/abs/2203.07618) [find](https://scholar.google.com/scholar?q=J%2E+Lee%2C+T%2E+Le%2C+J%2E+Chen%2C+and+D%2E+Lee%2C+%E2%80%9CDo+language+models+plagiarize%3F%E2%80%9D+arXiv+preprint+arXiv%3A2203%2E07618+%5Bcs%2ECL%5D%2C+2022%2E+%E2%80%94+%5BarXiv%3A2203%2E07618%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2203%2E07618%29)
37. E. Lehman, S. Jain, K. Pichotta, Y. Goldberg, and B. C. Wallace, “Does BERT pretrained on clinical notes reveal sensitive data?” arXiv preprint arXiv:2104.07762 [cs.CL], 2021. [find](https://scholar.google.com/scholar?q=E%2E+Lehman%2C+S%2E+Jain%2C+K%2E+Pichotta%2C+Y%2E+Goldberg%2C+and+B%2E+C%2E+Wallace%2C+%E2%80%9CDoes+BERT+pretrained+on+clinical+notes+reveal+sensitive+data%3F%E2%80%9D+arXiv+preprint+arXiv%3A2104%2E07762+%5Bcs%2ECL%5D%2C+2021%2E)
38. X. Li, F. Tramer, P. Liang, and T. Hashimoto, “Large language models can be strong differentially private learners,” in 10th International Conference on Learning Representations (ICLR). OpenReview.net, 2022. [find](https://scholar.google.com/scholar?q=X%2E+Li%2C+F%2E+Tramer%2C+P%2E+Liang%2C+and+T%2E+Hashimoto%2C+%E2%80%9CLarge+language+models+can+be+strong+differentially+private+learners%2C%E2%80%9D+in+10th+International+Conference+on+Learning+Representations+%28ICLR%29%2E+OpenReview%2Enet)
39. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “[RoBERTa: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/),” arXiv preprint arXiv:1907.11692 [cs.CL], 2019.
40. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in 7th International Conference on Learning Representations, ICLR. OpenReview.net, 2019. [find](https://scholar.google.com/scholar?q=I%2E+Loshchilov+and+F%2E+Hutter%2C+%E2%80%9CDecoupled+weight+decay+regularization%2C%E2%80%9D+in+7th+International+Conference+on+Learning+Representations%2C+ICLR%2E+OpenReview%2Enet%2C+2019%2E)
41. N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Béguelin, “[Analyzing leakage of personally identifiable information in language models](https://md.research.elamur.ai/free/en/papers/lukas-2023-analyzing-leakage-personally-identifiable-information/),” in 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023.
42. R. T. McCoy, P. Smolensky, T. Linzen, J. Gao, and A. Celikyilmaz, “[How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://md.research.elamur.ai/free/en/papers/mccoy-2023-how-much-do-language-models/),” arXiv preprint arXiv:2111.09509 [cs.CL], 2021.
43. B. McMahan and A. Thakurta, “Federated learning with formal differential privacy guarantees,” Google, Tech. Rep., 2022. [Online]. Available: [https://ai.googleblog.com/2022/02/federated-learning-with-formal.html](https://ai.googleblog.com/2022/02/federated-learning-with-formal.html) [↗](https://ai.googleblog.com/2022/02/federated-learning-with-formal.html)
44. N. Miao, H. Zhou, L. Mou, R. Yan, and L. Li, “CGMH: Constrained sentence generation by metropolis-hastings sampling,” in AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 6834–6842. [find](https://scholar.google.com/scholar?q=N%2E+Miao%2C+H%2E+Zhou%2C+L%2E+Mou%2C+R%2E+Yan%2C+and+L%2E+Li%2C+%E2%80%9CCGMH%3A+Constrained+sentence+generation+by+metropolis%2Dhastings+sampling%2C%E2%80%9D+in+AAAI+Conference+on+Artificial+Intelligence%2C+vol%2E+33%2C+no%2E+01%2C+2019%2C+pp%2E+6834%E2%80%93684)
45. F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri, “[Quantifying privacy risks of masked language models using membership inference attacks](https://md.research.elamur.ai/free/en/papers/mireshghallah-2022-quantifying-privacy-risks-masked-language/),” arXiv preprint arXiv:2203.03929 [cs.LG], 2022.
46. F. Mireshghallah, A. Uniyal, T. Wang, D. Evans, and T. Berg-Kirkpatrick, “Memorization in NLP fine-tuning methods,” arXiv preprint arXiv:2205.12506 [cs.CL], 2022. — [arXiv:2205.12506](https://arxiv.org/abs/2205.12506) [find](https://scholar.google.com/scholar?q=F%2E+Mireshghallah%2C+A%2E+Uniyal%2C+T%2E+Wang%2C+D%2E+Evans%2C+and+T%2E+Berg%2DKirkpatrick%2C+%E2%80%9CMemorization+in+NLP+fine%2Dtuning+methods%2C%E2%80%9D+arXiv+preprint+arXiv%3A2205%2E12506+%5Bcs%2ECL%5D%2C+2022%2E+%E2%80%94+%5BarXiv%3A2205%2E12506%5D%28https%3A%2F%2Farxiv%2Eor)
47. D. Nadeau and S. Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes, vol. 30, no. 1, pp. 3–26, 2007. [find](https://scholar.google.com/scholar?q=D%2E+Nadeau+and+S%2E+Sekine%2C+%E2%80%9CA+survey+of+named+entity+recognition+and+classification%2C%E2%80%9D+Lingvisticae+Investigationes%2C+vol%2E+30%2C+no%2E+1%2C+pp%2E+3%E2%80%9326%2C+2007%2E)
48. M. Nadeem, A. Bethke, and S. Reddy, “[StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/),” in 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP. ACL, 2021, pp. 5356–5371.
49. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders et al., “[WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/),” arXiv preprint arXiv:2112.09332 [cs.CL], 2021.
50. R. Parikh, C. Dupuy, and R. Gupta, “Canary extraction in natural language understanding models,” in 60th Annual Meeting of the Association for Computational Linguistics, ACL. ACL, 2022, pp. 552–560. [find](https://scholar.google.com/scholar?q=R%2E+Parikh%2C+C%2E+Dupuy%2C+and+R%2E+Gupta%2C+%E2%80%9CCanary+extraction+in+natural+language+understanding+models%2C%E2%80%9D+in+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+ACL%2E+ACL%2C+2022%2C+pp%2E+552%E2%80%93560%2E)
51. I. Pilán, P. Lison, L. Øvrelid, A. Papadopoulou, D. Sánchez, and M. Batet, “[The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization](https://md.research.elamur.ai/free/en/papers/pilan-2022-text-anonymization-benchmark-tab-dedicated/),” Computational Linguistics, vol. 48, no. 4, pp. 1053–1101, 2022.
52. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners.” [Online]. Available: [https://cdn.openai.com/better-language-models/language](https://cdn.openai.com/better-language-models/language) models are unsupervised multitask learners.pdf [↗](https://cdn.openai.com/better-language-models/language)
53. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu et al., “[Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/).” J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, 2020.
54. L. Rocher, J. M. Hendrickx, and Y.-A. De Montjoye, “Estimating the success of re-identifications in incomplete datasets using generative models,” Nature communications, vol. 10, no. 1, pp. 1–9, 2019. [find](https://scholar.google.com/scholar?q=L%2E+Rocher%2C+J%2E+M%2E+Hendrickx%2C+and+Y%2E%2DA%2E+De+Montjoye%2C+%E2%80%9CEstimating+the+success+of+re%2Didentifications+in+incomplete+datasets+using+generative+models%2C%E2%80%9D+Nature+communications%2C+vol%2E+10%2C+no%2E+1%2C+pp%2E+1%E2%80%939%2C+2019%2E)
55. P. Samarati and L. Sweeney, “Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression,” Computer Science Laboratory, SRI International, Tech. Rep., 1998. [Online]. Available: [http://www.csl.sri.com/](http://www.csl.sri.com/) papers/sritr-98-04/ [↗](http://www.csl.sri.com/)
56. R. Sennrich, B. Haddow, and A. Birch, “[Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/),” arXiv preprint arXiv:1508.07909 [cs.CL], 2015.
57. W. Shi, A. Cui, E. Li, R. Jia, and Z. Yu, “Selective differential privacy for language modeling,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACACL-HLT. ACL, 2022, pp. 2848–2859. [find](https://scholar.google.com/scholar?q=W%2E+Shi%2C+A%2E+Cui%2C+E%2E+Li%2C+R%2E+Jia%2C+and+Z%2E+Yu%2C+%E2%80%9CSelective+differential+privacy+for+language+modeling%2C%E2%80%9D+in+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+La)
58. R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “[Membership inference attacks against machine learning models](https://md.research.elamur.ai/free/en/papers/shokri-2016-membership-inference-attacks-against-machine/),” in 37th IEEE symposium on security and privacy (S&P). IEEE, 2017, pp. 3–18.
59. P. Stock, I. Shilov, I. Mironov, and A. Sablayrolles, “Defending against reconstruction attacks with Rényi differential privacy,” arXiv preprint arXiv:2202.07623 [cs.LG], 2022. — [arXiv:2202.07623](https://arxiv.org/abs/2202.07623) [find](https://scholar.google.com/scholar?q=P%2E+Stock%2C+I%2E+Shilov%2C+I%2E+Mironov%2C+and+A%2E+Sablayrolles%2C+%E2%80%9CDefending+against+reconstruction+attacks+with+Re%CC%81nyi+differential+privacy%2C%E2%80%9D+arXiv+preprint+arXiv%3A2202%2E07623+%5Bcs%2ELG%5D%2C+2022%2E+%E2%80%94+%5BarXiv%3A2202%2E07623%5D%28h)
60. K. Tirumala, A. H. Markosyan, L. Zettlemoyer, and A. Aghajanyan, “[Memorization without overfitting: Analyzing the training dynamics of large language models](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/),” in Advances in neural information processing systems (NeurIPS), vol. 35. Curran Associates, Inc., 2022, pp. 38 274–38 290.
61. T. Vakili and H. Dalianis, “Are clinical BERT models privacy preserving? The difficulty of extracting patient-condition associations.” in AAAI 2021 Fall Symposium on Human Partnership with Medical AI: Design, Operationalization, and Ethics (AAAI-HUMAN). CEUR-WS.org, 2021. [find](https://scholar.google.com/scholar?q=T%2E+Vakili+and+H%2E+Dalianis%2C+%E2%80%9CAre+clinical+BERT+models+privacy+preserving%3F+The+difficulty+of+extracting+patient%2Dcondition+associations%2E%E2%80%9D+in+AAAI+2021+Fall+Symposium+on+Human+Partnership+with+Medical+AI%3A)
62. T. Vakili, A. Lamproudis, A. Henriksson, and H. Dalianis, “Downstream task performance of BERT models pre-trained using automatically de-identified clinical data,” in 13th Language Resources and Evaluation Conference, LREC. ACL, 2022, pp. 4245–4252. [find](https://scholar.google.com/scholar?q=T%2E+Vakili%2C+A%2E+Lamproudis%2C+A%2E+Henriksson%2C+and+H%2E+Dalianis%2C+%E2%80%9CDownstream+task+performance+of+BERT+models+pre%2Dtrained+using+automatically+de%2Didentified+clinical+data%2C%E2%80%9D+in+13th+Language+Resources+and+Evalu)
63. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” Advances in neural information processing systems (NeurIPS), vol. 30, p. 6000–6010, 2017.
64. L. Wutschitz, H. A. Inan, and A. Manoel, “dp-transformers: Training transformer models with differential privacy.” [Online]. Available: [https://www.microsoft.com/en-us/research/project/dp-transformers](https://www.microsoft.com/en-us/research/project/dp-transformers) [↗](https://www.microsoft.com/en-us/research/project/dp-transformers)
65. S. Yeom, I. Giacomelli, A. Menaged, M. Fredrikson, and S. Jha, “Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning,” Journal of Computer Security, vol. 28, no. 1, pp. 35–70, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Yeom%2C+I%2E+Giacomelli%2C+A%2E+Menaged%2C+M%2E+Fredrikson%2C+and+S%2E+Jha%2C+%E2%80%9COverfitting%2C+robustness%2C+and+malicious+algorithms%3A+A+study+of+potential+causes+of+privacy+risk+in+machine+learning%2C%E2%80%9D+Journal+of+Computer)
66. A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao et al., “Opacus: User-friendly differential privacy library in PyTorch,” arXiv preprint arXiv:2109.12298 [cs.LG], 2021. — [arXiv:2109.12298](https://arxiv.org/abs/2109.12298) [find](https://scholar.google.com/scholar?q=A%2E+Yousefpour%2C+I%2E+Shilov%2C+A%2E+Sablayrolles%2C+D%2E+Testuggine%2C+K%2E+Prasad%2C+M%2E+Malek%2C+J%2E+Nguyen%2C+S%2E+Ghosh%2C+A%2E+Bharadwaj%2C+J%2E+Zhao+et+al%2E%2C+%E2%80%9COpacus%3A+User%2Dfriendly+differential+privacy+library+in+PyTorch%2C%E2%80%9D+arXiv)
67. D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz, S. Yekhanin, and H. Zhang, “Differentially private fine-tuning of language models,” in 10th International Conference on Learning Representations (ICLR). OpenReview.net, 2022. [find](https://scholar.google.com/scholar?q=D%2E+Yu%2C+S%2E+Naik%2C+A%2E+Backurs%2C+S%2E+Gopi%2C+H%2E+A%2E+Inan%2C+G%2E+Kamath%2C+J%2E+Kulkarni%2C+Y%2E+T%2E+Lee%2C+A%2E+Manoel%2C+L%2E+Wutschitz%2C+S%2E+Yekhanin%2C+and+H%2E+Zhang%2C+%E2%80%9CDifferentially+private+fine%2Dtuning+of+language+models%2C%E2%80%9D+in+10th)
68. S. Zanella-Béguelin, L. Wutschitz, S. Tople, V. Rühle, A. Paverd, O. Ohrimenko, B. Köpf, and M. Brockschmidt, “Analyzing information leakage of updates to natural language models,” in 27th ACM SIGSAC Conference on Computer and Communications Security, 2020, pp. 363–375. [find](https://scholar.google.com/scholar?q=S%2E+Zanella%2DBe%CC%81guelin%2C+L%2E+Wutschitz%2C+S%2E+Tople%2C+V%2E+Ru%CC%88hle%2C+A%2E+Paverd%2C+O%2E+Ohrimenko%2C+B%2E+Ko%CC%88pf%2C+and+M%2E+Brockschmidt%2C+%E2%80%9CAnalyzing+information+leakage+of+updates+to+natural+language+models%2C%E2%80%9D+in+27th+ACM+SIGS)
69. R. Zhang, S. Hidano, and F. Koushanfar, “Text revealer: Private text reconstruction via model inversion attacks against transformers,” arXiv preprint arXiv:2209.10505 [cs.CL], 2022. — [arXiv:2209.10505](https://arxiv.org/abs/2209.10505) [find](https://scholar.google.com/scholar?q=R%2E+Zhang%2C+S%2E+Hidano%2C+and+F%2E+Koushanfar%2C+%E2%80%9CText+revealer%3A+Private+text+reconstruction+via+model+inversion+attacks+against+transformers%2C%E2%80%9D+arXiv+preprint+arXiv%3A2209%2E10505+%5Bcs%2ECL%5D%2C+2022%2E+%E2%80%94+%5BarXiv%3A2209%2E1050)
70. X. Zhao, L. Li, and Y.-X. Wang, “Provably confidential language modelling,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACACL-HLT. ACL, 2022, pp. 943–955. [find](https://scholar.google.com/scholar?q=X%2E+Zhao%2C+L%2E+Li%2C+and+Y%2E%2DX%2E+Wang%2C+%E2%80%9CProvably+confidential+language+modelling%2C%E2%80%9D+in+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies%2C+N)

*Updated 2 November 2026*