---
title: "Ita-Lib at SVELA — paper by Yassine et al. (2026)"
description: "The organizers' benchmark builds on the FAME framework: fictional people with synthetic biographies… 20 references, cited by 1 paper here."
authors: ["Ali Yassine", "Hadi Ibrahim", "Luca Cagliero"]
year: 2026
venue: "Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)"
source: "https://aclanthology.org/2026.evalita-1.44/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yassine-2026-ita-lib-svela-detecting-forgotten/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yassine-2026-ita-lib-svela-detecting-forgotten/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/) / Ita-Lib at SVELA: Detecting the Forgotten –- Representation-Based Approach for Verifying Machine Unlearning

# Ita-Lib at SVELA: Detecting the Forgotten –- Representation-Based Approach for Verifying Machine Unlearning

**Authors:** [Ali Yassine](https://md.research.elamur.ai/free/en/authors/ali-yassine/), Hadi Ibrahim and [Luca Cagliero](https://md.research.elamur.ai/free/en/authors/luca-cagliero/) · **Year:** 2026 · **Venue:** [Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)](https://md.research.elamur.ai/free/en/venues/evalita/) ([Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop 2026](https://md.research.elamur.ai/free/en/venues/evalita/2026/)) · **Publisher:** [CEUR Workshop Proceedings](https://md.research.elamur.ai/free/en/publishers/ceur-workshop-proceedings/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.evalita-1.44/) · [PDF](https://aclanthology.org/2026.evalita-1.44.pdf) · [OpenAlex](https://openalex.org/W7217794504)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/yassine-2026-ita-lib-svela-detecting-forgotten/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses post-hoc auditing of machine unlearning in LLMs. The goal is to decide whether a model still holds knowledge about an entity, has had it erased, or never learned it, without access to training data or to the erasure procedure. Politecnico di Torino submitted this work to Task 1 (entity-level unlearning) of SVELA, a challenge hosted within EVALITA 2026. The organizers' benchmark builds on the FAME framework: fictional people with synthetic biographies and automatically generated QA pairs in Italian, Spanish, French and German. Because the people are fictional, any knowledge a model shows comes only from controlled fine-tuning. The authors pool hidden states from late transformer layers and train a shallow classifier that labels each identity as Retain, Forget or Test (unseen). Retained identities are detected reliably across model sizes and unlearning methods, but forgotten ones are hard to separate from unseen ones. The authors attribute this partly to successful unlearning and partly to the small Forget class. The work highlights a key tension for GDPR-style auditing: the stronger the erasure, the weaker the signal left for an external verifier.

## Key points
- Data: 800 fictional identities across four languages, with 20 QA pairs each (16,000 samples). The labeled training split has 882 rows (576 Retain, 114 Forget, 192 Test), and the unlabeled test set has 4,420 rows.
- Class imbalance: Forget makes up only about 13% of the training data, which pushes the classifier toward the Retain and Test classes.
- Method: the approach is model-agnostic and post-hoc. It aggregates late-layer hidden representations and feeds them to a lightweight probe, deliberately kept shallow so it reads existing signals rather than re-learning.
- Baseline: a logit-based baseline using mean logit distributions reaches higher accuracy but only 0.281 Macro F1, because it is biased toward Retain. The authors argue that hidden-state features add signal the logits miss.
- Observed trends: verification works slightly better on smaller models, and some unlearning algorithms leave more detectable traces than others. Proposed future work includes multi-layer aggregation, paraphrase consistency, logit uncertainty measures and Task 2 (fact-level verification).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Politecnico di Torino](https://md.research.elamur.ai/free/en/universities/politecnico-di-torino/) — University, Turin, Italy · 11 works · 18 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Ali Yassine, Hadi Ibrahim and Luca Cagliero. 2026. Ita-Lib at SVELA: Detecting the Forgotten –- Representation-Based Approach for Verifying Machine Unlearning. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yassine2026ita,
  title = {{Ita-Lib at SVELA: Detecting the Forgotten –- Representation-Based Approach for Verifying Machine Unlearning}},
  author = {Ali Yassine, Hadi Ibrahim and Luca Cagliero},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.44/}
}
```

- [← previous It's All About the Confidence: An Unsupervised Approach for Multilingual Historical Entity Linking using Large Language Models](https://md.research.elamur.ai/free/en/papers/santini-2026-it-s-all-about-confidence/)
- [next → Iterative Structured Pruning for Large Language Models with Multi-Domain Calibration](https://md.research.elamur.ai/free/en/papers/wu-2026-iterative-structured-pruning-large-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/)

## References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. European Parliament and Council of the European Union, Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation… [find](https://scholar.google.com/scholar?q=European+Parliament+and+Council+of+the+European+Union%2C+Regulation+%28EU%29+2016%2F679+of+the+European+Parliament+and+of+the+Council+of+27+April+2016+on+the+protection+of+natural+persons+with+regard+to+the+p)
2. A. Mantelero, The EU proposal for a general data protection regulation and the roots of the ‘right to be Forgotten’, Computer Law & Security Review 29 (2013) 229–235. [find](https://scholar.google.com/scholar?q=A%2E+Mantelero%2C+The+EU+proposal+for+a+general+data+protection+regulation+and+the+roots+of+the+%E2%80%98right+to+be+Forgotten%E2%80%99%2C+Computer+Law+%26+Security+Review+29+%282013%29+229%E2%80%93235%2E)
3. European Parliament and Council of the European Union, Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Official Journal of the European Union, L Series, 2024. URL: https://e… [find](https://scholar.google.com/scholar?q=European+Parliament+and+Council+of+the+European+Union%2C+Regulation+%28EU%29+2024%2F1689+of+the+European+Parliament+and+of+the+Council+of+13+June+2024+laying+down+harmonised+rules+on+artificial+intelligence+%28)
4. C. Savelli, M. La Quatra, A. Koudounas, F. Giobergia, [Svela at evalita 2026: Overview of the selective verification of erasure from llm answers task](https://md.research.elamur.ai/free/en/papers/savelli-2026-svela-evalita-2026-overview-selective/), in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org,…
5. F. Cutugno, A. Miaschi, A. P. Aprosio, G. Rambelli, L. Siciliani, M. A. Stranisci, Evalita 2026: Overview of the 9th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for I… [find](https://scholar.google.com/scholar?q=F%2E+Cutugno%2C+A%2E+Miaschi%2C+A%2E+P%2E+Aprosio%2C+G%2E+Rambelli%2C+L%2E+Siciliani%2C+M%2E+A%2E+Stranisci%2C+Evalita+2026%3A+Overview+of+the+9th+evaluation+campaign+of+natural+language+processing+and+speech+tools+for+italian%2C+in)
6. L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choopani, H. Faghri, A. Higbee, M. Jagielski, A. S. Jia, V. Paneerhelvam, J. Paige, A. Ali, Machine unlearning, in: 2021 IEEE Symposium on Security and Privacy (SP), IEEE, 2021, pp. 141–159. [find](https://scholar.google.com/scholar?q=L%2E+Bourtoule%2C+V%2E+Chandrasekaran%2C+C%2E+A%2E+Choquette%2DChoopani%2C+H%2E+Faghri%2C+A%2E+Higbee%2C+M%2E+Jagielski%2C+A%2E+S%2E+Jia%2C+V%2E+Paneerhelvam%2C+J%2E+Paige%2C+A%2E+Ali%2C+Machine+unlearning%2C+in%3A+2021+IEEE+Symposium+on+Security+and)
7. A. Thudi, G. Deza, V. Chandrasekaran, N. Papernot, Unrolling SGD: Understanding Factors Influencing Machine Unlearning , in: 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), IEEE Computer Society, Los Alamitos, CA, USA, 2022, pp. 303–319. URL: [https://doi.ieeecomputersociety.org/1…](https://doi.ieeecomputersociety.org/1…) — [doi:10.1109/eurosp53844.2022.00027](https://doi.org/10.1109/eurosp53844.2022.00027) [↗](https://doi.ieeecomputersociety.org/1…)
8. T. T. Nguyen, T. T. Huynh, Z. Ren, P. L. Nguyen, A. W.-C. Liew, H. Yin, Q. V. H. Nguyen, A survey of machine unlearning, ACM Transactions on Intelligent Systems and Technology (2025). [find](https://scholar.google.com/scholar?q=T%2E+T%2E+Nguyen%2C+T%2E+T%2E+Huynh%2C+Z%2E+Ren%2C+P%2E+L%2E+Nguyen%2C+A%2E+W%2E%2DC%2E+Liew%2C+H%2E+Yin%2C+Q%2E+V%2E+H%2E+Nguyen%2C+A+survey+of+machine+unlearning%2C+ACM+Transactions+on+Intelligent+Systems+and+Technology+%282025%29%2E)
9. J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, M. Seo, Knowledge unlearning for mitigating privacy risks in language models, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A… [find](https://scholar.google.com/scholar?q=J%2E+Jang%2C+D%2E+Yoon%2C+S%2E+Yang%2C+S%2E+Cha%2C+M%2E+Lee%2C+L%2E+Logeswaran%2C+M%2E+Seo%2C+Knowledge+unlearning+for+mitigating+privacy+risks+in+language+models%2C+in%3A+A%2E+Rogers%2C+J%2E+Boyd%2DGraber%2C+N%2E+Okazaki+%28Eds%2E%29%2C+Proceedings+of)
10. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, J. Z. Kolter, [TOFU: A task of fictitious unlearning for LLMs](https://md.research.elamur.ai/free/en/papers/maini-2024-tofu-task-fictitious-unlearning-llms/), in: First Conference on Language Modeling, 2024. URL: [https://openreview.net/forum?](https://openreview.net/forum?) id=B41hNBoWLo. [↗](https://openreview.net/forum?)
11. R. Shokri, M. Stronati, C. Song, V. Shmatikov, [Membership inference attacks against machine learning models](https://md.research.elamur.ai/free/en/papers/shokri-2016-membership-inference-attacks-against-machine/), in: 2017 IEEE Symposium on Security and Privacy (SP), IEEE, 2017, pp. 3–18.
12. J. Hewitt, C. D. Manning, [A structural probe for finding syntax in word representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/), in: NAACL, 2019.
13. Y. Belinkov, J. Glass, [Analysis methods in neural language processing: A survey](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/), in: Transactions of the Association for Computational Linguistics, 2019.
14. A. Rogers, O. Kovaleva, A. Rumshisky, [A primer in bertology: What we know about how bert works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/), Transactions of the Association for Computational Linguistics (2020).
15. N. Carlini, et al., [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/), in: USENIX Security, 2021.
16. C. Song, A. Raghunathan, V. Shmatikov, Auditing data provenance in text-generation models, in: KDD, 2019. [find](https://scholar.google.com/scholar?q=C%2E+Song%2C+A%2E+Raghunathan%2C+V%2E+Shmatikov%2C+Auditing+data+provenance+in+text%2Dgeneration+models%2C+in%3A+KDD%2C+2019%2E)
17. I. Tenney, D. Das, E. Pavlick, [Bert rediscovers the classical nlp pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/), in: ACL, 2019.
18. Y. Cao, J. Yang, Towards making systems forget with machine unlearning, in: 2015 IEEE Symposium on Security and Privacy, IEEE, 2015, pp. 463–480. [find](https://scholar.google.com/scholar?q=Y%2E+Cao%2C+J%2E+Yang%2C+Towards+making+systems+forget+with+machine+unlearning%2C+in%3A+2015+IEEE+Symposium+on+Security+and+Privacy%2C+IEEE%2C+2015%2C+pp%2E+463%E2%80%93480%2E)
19. C. Savelli, M. La Quatra, A. Koudounas, F. Giobergia, [FAME: Fictional actors for multilingual erasure](https://md.research.elamur.ai/free/en/papers/savelli-2026-fame-fictional-actors-multilingual-erasure/), in: Proceedings of the Fifteenth Language Resources and Evaluation Conference, European Language Resources Association, 2026.
20. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/), in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, pp. 4171–4186.

*Updated 31 October 2026*