---
title: "Do LLMs Really Memorize Personally Identifiable Information?"
description: "The authors propose Cue-Resistant Memorization (CRM), an evaluation framework that conditions… 39 references, cited by 1 paper here."
authors: ["Xiaoyu Luo", "Yiyi Chen", "Qiongxiu Li", "Johannes Bjerva"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1560/"
doi: "10.18653/v1/2026.acl-long.1560"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/luo-2026-do-llms-really-memorize-personally/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/luo-2026-do-llms-really-memorize-personally/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) / Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Framework

# Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Framework

**Authors:** Xiaoyu Luo, [Yiyi Chen](https://md.research.elamur.ai/free/en/authors/yiyi-chen/), Qiongxiu Li and [Johannes Bjerva](https://md.research.elamur.ai/free/en/authors/johannes-bjerva/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1560/) · [PDF](https://aclanthology.org/2026.acl-long.1560.pdf) · [DOI 10.18653/v1/2026.acl-long.1560](https://doi.org/10.18653/v1/2026.acl-long.1560) · [OpenAlex](https://openalex.org/W7166875374)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Earlier studies often take an LLM's ability to reproduce personally identifiable information (PII) as proof that the model memorized it. This paper challenges that assumption. Many apparent "leaks" may come from the prompt itself: names, formatting conventions and character overlap can let a model guess an email address without having stored it. The authors propose Cue-Resistant Memorization (CRM), an evaluation framework that conditions memorization metrics on prompt–target overlap. Under CRM, success in low-cue conditions is a necessary requirement before memorization can be claimed. They apply it in a multilingual study across 32 languages and several mGPT models. The study covers verbatim prefix–suffix completion, associative reconstruction from an owner's name, cue-free generation and membership inference. Once overlap cues are controlled, reconstruction success falls to near zero. Log-likelihood follows cue strength equally for training and unseen samples. Free generation produces mostly generic addresses, and membership inference stays near chance. The findings suggest PII privacy risk in LLMs has been overestimated, and CRM offers a stricter protocol for privacy audits and for mitigation work such as model editing.

## Key points
- CRM is a cue-controlled framework that separates genuine memorization from pattern completion driven by surface cues. It is evaluated on mGPT3-13B, mGPT3-1.3B and mGPT2-560M across 32 languages.
- The study distinguishes three paradigms: verbatim, associative (e.g., name → email) and extractable (generic requests with no target context). Reconstruction successes occur almost only under high-cue conditions, and the effect is strongest for Latin-script languages.
- In cue-free generation, true positive rates for emails and phones range from roughly 0.23% to 0.77%. None of these hits overlap with PII recovered in the verbatim or associative settings.
- Eight membership inference attacks (including loss/perplexity, Zlib, reference-model, neighborhood and Min-K%/Min-K%++) reach AUROC of only 0.50–0.60. English averages 0.525, and differences between languages are small.
- The code is published on GitHub (xiaoyuluoit97/mllm_pii_mia). The paper appears in the ACL 2026 proceedings and comes from Aalborg University.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Aalborg University](https://md.research.elamur.ai/free/en/universities/aalborg-university/) — University, Aalborg, Denmark · 12 works · 743 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Xiaoyu Luo, Yiyi Chen, Qiongxiu Li and Johannes Bjerva. 2026. Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Framework. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{luo2026do,
  title = {{Do LLMs Really Memorize Personally Identifiable Information? Revisiting PII Leakage with a Cue-Controlled Memorization Framework}},
  author = {Xiaoyu Luo, Yiyi Chen, Qiongxiu Li and Johannes Bjerva},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1560},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1560}
}
```

- [← previous Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset](https://md.research.elamur.ai/free/en/papers/plum-2026-do-llms-judge-distantly-supervised/)
- [next → Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?](https://md.research.elamur.ai/free/en/papers/yuan-2026-do-mixed-vendor-multi-agent/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/)

## References

39 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022a. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Steve+Chien%2C+Milad+Nasr%2C+Shuang+Song%2C+Andreas+Terzis%2C+and+Florian+Tramer%2E+2022a%2E+Membership+inference+attacks+from+first+principles%2E+In+2022+IEEE+symposium+on+security+and+privacy+%28S)
2. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022b. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Florian+Tramer%2C+and+Chiyuan+Zhang%2E+2022b%2E+Quantifying+memorization+across+neural+language+models%2E+In+The+Eleventh+International+Con)
3. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. [The secret sharer: Evaluating and testing unintended memorization in neural networks](https://md.research.elamur.ai/free/en/papers/carlini-2018-secret-sharer-evaluating-testing-unintended/). In 28th USENIX security symposium (USENIX security 19), pages 267–284.
4. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650…
5. Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. [Speak, memory: An archaeology of books known to chatgpt/gpt-4](https://md.research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7312–7327.
6. Yiyi Chen, Russa Biswas, Heather Lent, and Johannes Bjerva. 2025. Against all odds: Overcoming typology, script, and language confusion in multilingual embedding inversion attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23632–23641. [find](https://scholar.google.com/scholar?q=Yiyi+Chen%2C+Russa+Biswas%2C+Heather+Lent%2C+and+Johannes+Bjerva%2E+2025%2E+Against+all+odds%3A+Overcoming+typology%2C+script%2C+and+language+confusion+in+multilingual+embedding+inversion+attacks%2E+In+Proceedings+of+t)
7. Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. [Do membership inference attacks work on large language models?](https://md.research.elamur.ai/free/en/papers/duan-2024-membership-inference-attacks-work-large/) In Conference on Language Modeling (COLM).
8. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, and 1 others. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+and+1+others%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+)
9. Xiaoyu Luo, Yiyi Chen, Johannes Bjerva, and Qiongxiu Li. 2025. Shared path: Unraveling memorization in multilingual llms through language similarities. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), Suzhou, China. [find](https://scholar.google.com/scholar?q=Xiaoyu+Luo%2C+Yiyi+Chen%2C+Johannes+Bjerva%2C+and+Qiongxiu+Li%2E+2025%2E+Shared+path%3A+Unraveling+memorization+in+multilingual+llms+through+language+similarities%2E+In+Proceedings+of+the+2025+Conference+on+Empiric)
10. Jamie Hayes, Ilia Shumailov, Christopher A Choquette-Choo, Matthew Jagielski, George Kaissis, Katherine Lee, Milad Nasr, Sahra Ghalebikesabi, Niloofar Mireshghallah, Meenatchi Sundaram Mutu Selva Annamalai, and 1 others. 2025. Strong membership inference attacks on massive datasets and (moderately)… [find](https://scholar.google.com/scholar?q=Jamie+Hayes%2C+Ilia+Shumailov%2C+Christopher+A+Choquette%2DChoo%2C+Matthew+Jagielski%2C+George+Kaissis%2C+Katherine+Lee%2C+Milad+Nasr%2C+Sahra+Ghalebikesabi%2C+Niloofar+Mireshghallah%2C+Meenatchi+Sundaram+Mutu+Selva+Anna)
11. Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023. [Membership inference attacks against language models via neighbourhood comparison](https://md.research.elamur.ai/free/en/papers/mattern-2023-membership-inference-attacks-against-language/). In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada…
12. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628. [find](https://scholar.google.com/scholar?q=Jie+Huang%2C+Hanyin+Shao%2C+and+Kevin+Chen%2DChuan+Chang%2E+2022%2E+Are+large+pre%2Dtrained+language+models+leaking+your+personal+information%3F+arXiv+preprint+arXiv%3A2205%2E12628%2E)
13. Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. 2021. Membership inference attack susceptibility of clinical language models. arXiv preprint arXiv:2104.08305. — [arXiv:2104.08305](https://arxiv.org/abs/2104.08305) [find](https://scholar.google.com/scholar?q=Abhyuday+Jagannatha%2C+Bhanu+Pratap+Singh+Rawat%2C+and+Hong+Yu%2E+2021%2E+Membership+inference+attack+susceptibility+of+clinical+language+models%2E+arXiv+preprint+arXiv%3A2104%2E08305%2E+%E2%80%94+%5BarXiv%3A2104%2E08305%5D%28https%3A%2F%2F)
14. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. [Copyright violations and large language models](https://md.research.elamur.ai/free/en/papers/karamolegkou-2023-copyright-violations-large-language-models/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7403–7412.
15. Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2023. Propile: Probing privacy leakage in large language models. Advances in Neural Information Processing Systems, 36:20750–20762. [find](https://scholar.google.com/scholar?q=Siwon+Kim%2C+Sangdoo+Yun%2C+Hwaran+Lee%2C+Martin+Gubri%2C+Sungroh+Yoon%2C+and+Seong+Joon+Oh%2E+2023%2E+Propile%3A+Probing+privacy+leakage+in+large+language+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+3)
16. Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In European conference on machine learning, pages 217–226. Springer. [find](https://scholar.google.com/scholar?q=Bryan+Klimt+and+Yiming+Yang%2E+2004%2E+The+enron+corpus%3A+A+new+dataset+for+email+classification+research%2E+In+European+conference+on+machine+learning%2C+pages+217%E2%80%93226%2E+Springer%2E)
17. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+2021%2E+Deduplicating+training+data+makes+language+models+better%2E+arXiv+preprint+a)
18. Bo Li, Qinghua Zhao, and Lijie Wen. 2024. Rome: Memorization insights from text, logits and representation. arXiv preprint arXiv:2403.00510. — [arXiv:2403.00510](https://arxiv.org/abs/2403.00510) [find](https://scholar.google.com/scholar?q=Bo+Li%2C+Qinghua+Zhao%2C+and+Lijie+Wen%2E+2024%2E+Rome%3A+Memorization+insights+from+text%2C+logits+and+representation%2E+arXiv+preprint+arXiv%3A2403%2E00510%2E+%E2%80%94+%5BarXiv%3A2403%2E00510%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E00510%29)
19. Qiongxiu Li, Xiaoyu Luo, Yiyi Chen, and Johannes Bjerva. 2025. Trustworthy machine learning via memorization and the granular long-tail: A survey on interactions, tradeoffs, and beyond. arXiv preprint arXiv:2503.07501. — [arXiv:2503.07501](https://arxiv.org/abs/2503.07501) [find](https://scholar.google.com/scholar?q=Qiongxiu+Li%2C+Xiaoyu+Luo%2C+Yiyi+Chen%2C+and+Johannes+Bjerva%2E+2025%2E+Trustworthy+machine+learning+via+memorization+and+the+granular+long%2Dtail%3A+A+survey+on+interactions%2C+tradeoffs%2C+and+beyond%2E+arXiv+preprint)
20. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing leakage of personally identifiable information in language models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 346–363. IEEE. [find](https://scholar.google.com/scholar?q=Nils+Lukas%2C+Ahmed+Salem%2C+Robert+Sim%2C+Shruti+Tople%2C+Lukas+Wutschitz%2C+and+Santiago+Zanella%2DB%C3%A9guelin%2E+2023%2E+Analyzing+leakage+of+personally+identifiable+information+in+language+models%2E+In+2023+IEEE+Sympo)
21. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. [Scalable extraction of training data from (production) language models](https://md.research.elamur.ai/free/en/papers/nasr-2023-scalable-extraction-training-data-production/). arXiv preprint arXiv:2311.17035.
22. Esther Ploeger, Wessel Poelman, Miryam de Lhoneux, and Johannes Bjerva. 2024. What is “typological diversity” in NLP? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5681–5700, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Esther+Ploeger%2C+Wessel+Poelman%2C+Miryam+de+Lhoneux%2C+and+Johannes+Bjerva%2E+2024%2E+What+is+%E2%80%9Ctypological+diversity%E2%80%9D+in+NLP%3F+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Pro)
23. Esther Ploeger, Wessel Poelman, Andreas Holck Høeg-Petersen, Anders Schlichtkrull, Miryam de Lhoneux, and Johannes Bjerva. 2025. [A principled framework for evaluating on typologically diverse languages](https://md.research.elamur.ai/free/en/papers/ploeger-2026-principled-framework-evaluating-typologically-diverse/). Computational Linguistics, pages 1–36.
24. QwenTeam. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
25. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67.
26. Elena Sofia Ruzzetti, Giancarlo A Xompero, Davide Venditti, and Fabio Massimo Zanzotto. 2025. Private memorization editing: Turning memorization into a defense to strengthen data privacy in large language models. arXiv preprint arXiv:2506.10024. [find](https://scholar.google.com/scholar?q=Elena+Sofia+Ruzzetti%2C+Giancarlo+A+Xompero%2C+Davide+Venditti%2C+and+Fabio+Massimo+Zanzotto%2E+2025%2E+Private+memorization+editing%3A+Turning+memorization+into+a+defense+to+strengthen+data+privacy+in+large+lang)
27. Tom Sander, Bargav Jayaraman, Mark Ibrahim, Kamalika Chaudhuri, and Chuan Guo. 2025. Rethinking the role of verbatim memorization in llm privacy. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tom+Sander%2C+Bargav+Jayaraman%2C+Mark+Ibrahim%2C+Kamalika+Chaudhuri%2C+and+Chuan+Guo%2E+2025%2E+Rethinking+the+role+of+verbatim+memorization+in+llm+privacy%2E+In+The+Thirty%2Dninth+Annual+Conference+on+Neural+Inform)
28. Ali Satvaty, Anna Visman, Dan Seidel, Suzan Verberne, and Fatih Turkmen. 2025. Memorization is language-sensitive: Analyzing memorization and inference risks of llms in a multilingual setting. In Proceedings of the First Workshop on Large Language Model Memorization (L2M2), pages 106–126. [find](https://scholar.google.com/scholar?q=Ali+Satvaty%2C+Anna+Visman%2C+Dan+Seidel%2C+Suzan+Verberne%2C+and+Fatih+Turkmen%2E+2025%2E+Memorization+is+language%2Dsensitive%3A+Analyzing+memorization+and+inference+risks+of+llms+in+a+multilingual+setting%2E+In+Proc)
29. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. Preprint, arXiv:2310.16789. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Anirudh+Ajith%2C+Mengzhou+Xia%2C+Yangsibo+Huang%2C+Daogao+Liu%2C+Terra+Blevins%2C+Danqi+Chen%2C+and+Luke+Zettlemoyer%2E+2023%2E+Detecting+pretraining+data+from+large+language+models%2E+Preprint%2C+arXiv%3A2310%2E)
30. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, and Tatiana Shavrina. 2024. [mgpt: Few-shot learners go multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/). Transactions of the Association for Computational Linguistics, 12:58–79.
31. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. [Membership inference attacks against machine learning models](https://md.research.elamur.ai/free/en/papers/shokri-2016-membership-inference-attacks-against-machine/). In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE.
32. Hedvig Skirgård, Hannah J Haynie, Damián E Blasi, Harald Hammarström, Jeremy Collins, Jay J Latarche, Jakob Lesage, Tobias Weber, Alena Witzlack-Makarevich, Sam Passmore, and 1 others. 2023. Grambank reveals the importance of genealogical constraints on linguistic diversity and highlights the impact… [find](https://scholar.google.com/scholar?q=Hedvig+Skirg%C3%A5rd%2C+Hannah+J+Haynie%2C+Dami%C3%A1n+E+Blasi%2C+Harald+Hammarstr%C3%B6m%2C+Jeremy+Collins%2C+Jay+J+Latarche%2C+Jakob+Lesage%2C+Tobias+Weber%2C+Alena+Witzlack%2DMakarevich%2C+Sam+Passmore%2C+and+1+others%2E+2023%2E+Grambank+)
33. Alisha Srivastava, Emir Korukluoglu, Minh Nhat Le, Duyen Tran, Chau Minh Pham, Marzena Karpinska, and Mohit Iyyer. 2025. Owl: Probing cross-lingual recall of memorized texts via world literature. arXiv preprint arXiv:2505.22945. [find](https://scholar.google.com/scholar?q=Alisha+Srivastava%2C+Emir+Korukluoglu%2C+Minh+Nhat+Le%2C+Duyen+Tran%2C+Chau+Minh+Pham%2C+Marzena+Karpinska%2C+and+Mohit+Iyyer%2E+2025%2E+Owl%3A+Probing+cross%2Dlingual+recall+of+memorized+texts+via+world+literature%2E+arXi)
34. Zhixing Tan, Xiangwen Zhang, Shuo Wang, and Yang Liu. 2021. Msp: Multi-stage prompting for making pre-trained language models better translators. arXiv preprint arXiv:2110.06609. [find](https://scholar.google.com/scholar?q=Zhixing+Tan%2C+Xiangwen+Zhang%2C+Shuo+Wang%2C+and+Yang+Liu%2E+2021%2E+Msp%3A+Multi%2Dstage+prompting+for+making+pre%2Dtrained+language+models+better+translators%2E+arXiv+preprint+arXiv%3A2110%2E06609%2E)
35. Davide Venditti, Elena Sofia Ruzzetti, Giancarlo A Xompero, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, and Fabio Massimo Zanzotto. 2024. Enhancing data privacy in large language models through private association editing. arXiv preprint arXiv:2406.18221. — [arXiv:2406.18221](https://arxiv.org/abs/2406.18221) [find](https://scholar.google.com/scholar?q=Davide+Venditti%2C+Elena+Sofia+Ruzzetti%2C+Giancarlo+A+Xompero%2C+Cristina+Giannone%2C+Andrea+Favalli%2C+Raniero+Romagnoli%2C+and+Fabio+Massimo+Zanzotto%2E+2024%2E+Enhancing+data+privacy+in+large+language+models+thro)
36. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/](https://github.com/kingoflolz/) mesh-transformer-jax. [↗](https://github.com/kingoflolz/)
37. Chengkun Wei, Minghu Zhao, Zhikun Zhang, Min Chen, Wenlong Meng, Bo Liu, Yuan Fan, and Wenzhi Chen. 2023. Dpmlbench: Holistic evaluation of differentially private machine learning. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2621–2635. [find](https://scholar.google.com/scholar?q=Chengkun+Wei%2C+Minghu+Zhao%2C+Zhikun+Zhang%2C+Min+Chen%2C+Wenlong+Meng%2C+Bo+Liu%2C+Yuan+Fan%2C+and+Wenzhi+Chen%2E+2023%2E+Dpmlbench%3A+Holistic+evaluation+of+differentially+private+machine+learning%2E+In+Proceedings+of+t)
38. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, and 1 others. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=BigScience+Workshop%2C+Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+and+1+others%2E+2022%2E+Bloom%3A+A+17)
39. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). arXiv preprint arXiv:2010.11934.

*Updated 31 October 2026*