---
title: "SVELA at EVALITA 2026 — paper by Savelli et al. (2026)"
description: "Participants had to build black-box or white-box verifiers that label information as retain, forget or… 22 references, cited by 6 papers here."
authors: ["Claudio Savelli", "Moreno La Quatra", "Alkis Koudounas", "Flavio Giobergia"]
year: 2026
venue: "Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)"
source: "https://aclanthology.org/2026.evalita-1.42/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/savelli-2026-svela-evalita-2026-overview-selective/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/savelli-2026-svela-evalita-2026-overview-selective/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / SVELA at EVALITA 2026: Overview of the Selective Verification of Erasure from LLM Answers Task

# SVELA at EVALITA 2026: Overview of the Selective Verification of Erasure from LLM Answers Task

**Authors:** [Claudio Savelli](https://md.research.elamur.ai/free/en/authors/claudio-savelli/), [Moreno La Quatra](https://md.research.elamur.ai/free/en/authors/moreno-la-quatra/), [Alkis Koudounas](https://md.research.elamur.ai/free/en/authors/alkis-koudounas/) and [Flavio Giobergia](https://md.research.elamur.ai/free/en/authors/flavio-giobergia/) · **Year:** 2026 · **Venue:** [Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)](https://md.research.elamur.ai/free/en/venues/evalita/) ([Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop 2026](https://md.research.elamur.ai/free/en/venues/evalita/2026/)) · **Publisher:** [CEUR Workshop Proceedings](https://md.research.elamur.ai/free/en/publishers/ceur-workshop-proceedings/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.evalita-1.42/) · [PDF](https://aclanthology.org/2026.evalita-1.42.pdf) · [OpenAlex](https://openalex.org/W7216860565)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/savelli-2026-svela-evalita-2026-overview-selective/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
SVELA, a shared task at EVALITA 2026, addresses how to audit machine unlearning. The question is whether a Large Language Model that has been through an unlearning procedure has actually lost targeted knowledge, which matters for GDPR-style deletion requests. The organizers fine-tuned LLMs on synthetic actor biographies from the FAME dataset, which covers Italian, Spanish, French and German. They then applied undisclosed unlearning methods.

Participants had to build black-box or white-box verifiers that label information as retain, forget or never-used. This was done at two levels:
- **Subtask 1:** whole identities.
- **Subtask 2:** single question-answer facts.

Systems were submitted as runnable code and executed on both public and hidden models, so generalization was tested fairly. Scoring used macro-F1.

Key findings:
- Fact-level verification was clearly harder than identity-level verification.
- Retained knowledge was easy to spot.
- Forgotten and never-seen content were hard to separate, because unlearning makes a model respond as if it never saw the data.

Overall performance was modest, so reliable unlearning verification remains an open problem.

## Key points
- Participation: eight teams registered, four wrote system papers, and there were more than fifty valid submissions across both subtasks.
- Data: each FAME identity has 20 atomic facts, each mapped to one QA pair, in four groups: Biography (5), Career (7), Achievements (3) and Personal (5).
- Models and methods: verifiers were evaluated on 1B and 3B models, with 3B giving small but steady macro-F1 gains. The unlearning methods included GA, GD, KLM and PO.
- Top result: the team priyam_saha17 ranked first in both subtasks for both model sizes.
- Method comparison: PO (preference-based unlearning) gave slightly more detectable forgetting, likely because it teaches the model to abstain with answers like "I don't know."

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Politecnico di Torino](https://md.research.elamur.ai/free/en/universities/politecnico-di-torino/) — University, Turin, Italy · 11 works · 18 citations
- [Università degli Studi di Enna Kore](https://md.research.elamur.ai/free/en/universities/universita-degli-studi-di-enna-kore/) — University, Enna, Italy · 2 works · 6 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Claudio Savelli, Moreno La Quatra, Alkis Koudounas and Flavio Giobergia. 2026. SVELA at EVALITA 2026: Overview of the Selective Verification of Erasure from LLM Answers Task. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{savelli2026svela,
  title = {{SVELA at EVALITA 2026: Overview of the Selective Verification of Erasure from LLM Answers Task}},
  author = {Claudio Savelli, Moreno {La Quatra}, Alkis Koudounas and Flavio Giobergia},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.42/}
}
```

- [← previous SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models](https://md.research.elamur.ai/free/en/papers/wu-2026-superwriter-reflection-driven-long-form/)
- [next → The Evolution of Thought: Tracking LLM Overthinking via Reasoning Dynamics Analysis](https://md.research.elamur.ai/free/en/papers/wei-2026-evolution-thought-tracking-llm-overthinking/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, C. Zhang, Quantifying memorization across neural language models, in: The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=N%2E+Carlini%2C+D%2E+Ippolito%2C+M%2E+Jagielski%2C+K%2E+Lee%2C+F%2E+Tramer%2C+C%2E+Zhang%2C+Quantifying+memorization+across+neural+language+models%2C+in%3A+The+Eleventh+International+Conference+on+Learning+Representations%2C+2022%2E)
2. A. Mantelero, The eu proposal for a general data protection regulation and the roots of the ‘right to be forgotten’, Computer Law & Security Review 29 (2013) 229–235. [find](https://scholar.google.com/scholar?q=A%2E+Mantelero%2C+The+eu+proposal+for+a+general+data+protection+regulation+and+the+roots+of+the+%E2%80%98right+to+be+forgotten%E2%80%99%2C+Computer+Law+%26+Security+Review+29+%282013%29+229%E2%80%93235%2E)
3. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/), in: Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022, pp. 30016–300…
4. M. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, N. Papernot, Machine unlearning, in: Proceedings of the IEEE Symposium on Security and Privacy (SP), 2021, pp. 141–159. doi: 10.1109/SP40001.2021.00018 . — [doi:10.1109/sp40001.2021.00018](https://doi.org/10.1109/sp40001.2021.00018) [find](https://scholar.google.com/scholar?q=M%2E+Bourtoule%2C+V%2E+Chandrasekaran%2C+C%2E+A%2E+Choquette%2DChoo%2C+H%2E+Jia%2C+A%2E+Travers%2C+B%2E+Zhang%2C+D%2E+Lie%2C+N%2E+Papernot%2C+Machine+unlearning%2C+in%3A+Proceedings+of+the+IEEE+Symposium+on+Security+and+Privacy+%28SP%29%2C+2021%2C+)
5. S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, Y. Yao, C. Y. Liu, X. Xu, H. Li, et al., Rethinking machine unlearning for large language models, Nature Machine Intelligence (2025) 1–14. [find](https://scholar.google.com/scholar?q=S%2E+Liu%2C+Y%2E+Yao%2C+J%2E+Jia%2C+S%2E+Casper%2C+N%2E+Baracaldo%2C+P%2E+Hase%2C+Y%2E+Yao%2C+C%2E+Y%2E+Liu%2C+X%2E+Xu%2C+H%2E+Li%2C+et+al%2E%2C+Rethinking+machine+unlearning+for+large+language+models%2C+Nature+Machine+Intelligence+%282025%29+1%E2%80%9314%2E)
6. J. Yao, E. Chien, M. Du, X. Niu, T. Wang, Z. Cheng, X. Yue, [Machine unlearning of pre-trained large language models](https://md.research.elamur.ai/free/en/papers/yao-2024-machine-unlearning-pre-trained-large/), in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computat…
7. A. Koudounas, C. Savelli, F. Giobergia, E. Baralis, “Alexa, can you forget me?” Machine Unlearning Benchmark in Spoken Language Understanding, in: Interspeech 2025, 2025, pp. 1768–1772. doi: 10.21437/Interspeech.2025- 2607 . — [doi:10.21437/interspeech.2025-](https://doi.org/10.21437/interspeech.2025-) [find](https://scholar.google.com/scholar?q=A%2E+Koudounas%2C+C%2E+Savelli%2C+F%2E+Giobergia%2C+E%2E+Baralis%2C+%E2%80%9CAlexa%2C+can+you+forget+me%3F%E2%80%9D+Machine+Unlearning+Benchmark+in+Spoken+Language+Understanding%2C+in%3A+Interspeech+2025%2C+2025%2C+pp%2E+1768%E2%80%931772%2E+doi%3A+10%2E21437%2F)
8. D. Andrea, S. Claudio, T. Gabriele, G. Flavio, B. Elena, G. Stilo, et al., How to make reproducible research in machine unlearning with erasure, in: Proceedings of the Thirty-Fourth International Joint Conference onArtificial Intelligence,{IJCAI-25}, 2025, pp. 11025–11029. [find](https://scholar.google.com/scholar?q=D%2E+Andrea%2C+S%2E+Claudio%2C+T%2E+Gabriele%2C+G%2E+Flavio%2C+B%2E+Elena%2C+G%2E+Stilo%2C+et+al%2E%2C+How+to+make+reproducible+research+in+machine+unlearning+with+erasure%2C+in%3A+Proceedings+of+the+Thirty%2DFourth+International+Join)
9. A. D’Angelo, C. Savelli, G. Tagliente, F. Giobergia, E. Baralis, G. Stilo, Erasure: A modular and extensible framework for machine unlearning, in: Proceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025, pp. 6346–6350. [find](https://scholar.google.com/scholar?q=A%2E+D%E2%80%99Angelo%2C+C%2E+Savelli%2C+G%2E+Tagliente%2C+F%2E+Giobergia%2C+E%2E+Baralis%2C+G%2E+Stilo%2C+Erasure%3A+A+modular+and+extensible+framework+for+machine+unlearning%2C+in%3A+Proceedings+of+the+34th+ACM+International+Conference+)
10. F. Cutugno, A. Miaschi, A. P. Aprosio, G. Rambelli, L. Siciliani, M. A. Stranisci, Evalita 2026: Overview of the 9th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for I… [find](https://scholar.google.com/scholar?q=F%2E+Cutugno%2C+A%2E+Miaschi%2C+A%2E+P%2E+Aprosio%2C+G%2E+Rambelli%2C+L%2E+Siciliani%2C+M%2E+A%2E+Stranisci%2C+Evalita+2026%3A+Overview+of+the+9th+evaluation+campaign+of+natural+language+processing+and+speech+tools+for+italian%2C+in)
11. C. Savelli, M. La Quatra, A. Koudounas, F. Giobergia, [FAME: Fictional actors for multilingual erasure](https://md.research.elamur.ai/free/en/papers/savelli-2026-fame-fictional-actors-multilingual-erasure/), in: Proceedings of the Fifteenth Language Resources and Evaluation Conference, European Language Resources Association, 2026.
12. A. Golatkar, A. Achille, S. Soatto, Eternal sunshine of the spotless net: Selective forgetting in deep networks, in: CVPR, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Golatkar%2C+A%2E+Achille%2C+S%2E+Soatto%2C+Eternal+sunshine+of+the+spotless+net%3A+Selective+forgetting+in+deep+networks%2C+in%3A+CVPR%2C+2020%2E)
13. D. Choi, D. Na, Towards machine unlearning benchmarks: Forgetting the personal identities in facial recognition systems, arXiv preprint arXiv:2311.02240 (2023). — [arXiv:2311.02240](https://arxiv.org/abs/2311.02240) [find](https://scholar.google.com/scholar?q=D%2E+Choi%2C+D%2E+Na%2C+Towards+machine+unlearning+benchmarks%3A+Forgetting+the+personal+identities+in+facial+recognition+systems%2C+arXiv+preprint+arXiv%3A2311%2E02240+%282023%29%2E+%E2%80%94+%5BarXiv%3A2311%2E02240%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
14. M. Kurmanji, P. Triantafillou, J. Hayes, E. Triantafillou, Towards unbounded machine unlearning, NeurIPS 36 (2024). [find](https://scholar.google.com/scholar?q=M%2E+Kurmanji%2C+P%2E+Triantafillou%2C+J%2E+Hayes%2C+E%2E+Triantafillou%2C+Towards+unbounded+machine+unlearning%2C+NeurIPS+36+%282024%29%2E)
15. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, J. Z. Kolter, [Tofu: A task of fictitious unlearning for llms](https://md.research.elamur.ai/free/en/papers/maini-2024-tofu-task-fictitious-unlearning-llms/), arXiv preprint arXiv:2401.06121 (2024).
16. H. Xu, T. Zhu, L. Zhang, W. Zhou, P. S. Yu, Machine unlearning: A survey, ACM Comput. Surv. 56 (2023). URL: https://doi.org/10.1145/3603620. doi: 10.1145/3603620 . — [doi:10.1145/3603620](https://doi.org/10.1145/3603620) [↗](https://doi.org/10.1145/3603620)
17. P. Saha, [priyam_saha17 at svela: A feature-centric pipeline for verifying selective forgetting in large language models](https://md.research.elamur.ai/free/en/papers/saha-2026-priyam-saha17-svela-feature-centric/), in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org, Bari, Italy, 2026.
18. E. Munis, M. Sabato, E. Bayat, A. Lolli, [Malto at svela: A specific-attention-head approach for membership inference attacks in llms unlearning evaluation](https://md.research.elamur.ai/free/en/papers/munis-2026-malto-svela-specific-attention-head/), in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEU…
19. A. Yassine, H. Ibrahim, L. Cagliero, Ita-lib at svela: Detecting the forgotten — representation-based approach for verifying machine unlearning, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), [CEUR.org, Bari…](https://md.research.elamur.ai/free/en/papers/yassine-2026-ita-lib-svela-detecting-forgotten/)
20. M. Berta, T. Cerquitelli, Eraserhead at svela: Detecting llm forgetting via logit-space statistics, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org, Bari, Italy, 2026. [find](https://scholar.google.com/scholar?q=M%2E+Berta%2C+T%2E+Cerquitelli%2C+Eraserhead+at+svela%3A+Detecting+llm+forgetting+via+logit%2Dspace+statistics%2C+in%3A+Proceedings+of+the+Ninth+Evaluation+Campaign+of+Natural+Language+Processing+and+Speech+Tools+for)
21. W. Shi, J. Lee, Y. Huang, S. Malladi, J. Zhao, A. Holtzman, D. Liu, L. Zettlemoyer, N. A. Smith, C. Zhang, Muse: Machine unlearning six-way evaluation for language models, arXiv preprint arXiv:2407.06460 (2024). [find](https://scholar.google.com/scholar?q=W%2E+Shi%2C+J%2E+Lee%2C+Y%2E+Huang%2C+S%2E+Malladi%2C+J%2E+Zhao%2C+A%2E+Holtzman%2C+D%2E+Liu%2C+L%2E+Zettlemoyer%2C+N%2E+A%2E+Smith%2C+C%2E+Zhang%2C+Muse%3A+Machine+unlearning+six%2Dway+evaluation+for+language+models%2C+arXiv+preprint+arXiv%3A2407%2E06)
22. C. Savelli, E. Munis, E. Bayat, A. Grieco, F. Giobergia, Malto at semeval-2025 task 4: Dual teachers for unlearning sensitive content in llms, in: Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), 2025, pp. 1747–1752. [find](https://scholar.google.com/scholar?q=C%2E+Savelli%2C+E%2E+Munis%2C+E%2E+Bayat%2C+A%2E+Grieco%2C+F%2E+Giobergia%2C+Malto+at+semeval%2D2025+task+4%3A+Dual+teachers+for+unlearning+sensitive+content+in+llms%2C+in%3A+Proceedings+of+the+19th+International+Workshop+on+Sem)

*Updated 1 November 2026*