---
title: "MALTO at SVELA — paper by Munis et al. (2026)"
description: "The paper addresses how to check whether machine unlearning in LLMs actually worked. 24 references, cited by 1 paper here."
authors: ["Evren A. Munis", "Mattia Sabato", "Erfan Bayat", "Andrea Lolli"]
year: 2026
venue: "Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)"
source: "https://aclanthology.org/2026.evalita-1.45/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/munis-2026-malto-svela-specific-attention-head/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/munis-2026-malto-svela-specific-attention-head/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/) / MALTO at SVELA: A Specific-Attention-Head Approach for Membership Inference Attacks in LLMs Unlearning Evaluation

# MALTO at SVELA: A Specific-Attention-Head Approach for Membership Inference Attacks in LLMs Unlearning Evaluation

**Authors:** [Evren A. Munis](https://md.research.elamur.ai/free/en/authors/evren-ayberk-munis/), Mattia Sabato, Erfan Bayat and [Andrea Lolli](https://md.research.elamur.ai/free/en/authors/andrea-lolli/) · **Year:** 2026 · **Venue:** [Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)](https://md.research.elamur.ai/free/en/venues/evalita/) ([Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop 2026](https://md.research.elamur.ai/free/en/venues/evalita/2026/)) · **Publisher:** [CEUR Workshop Proceedings](https://md.research.elamur.ai/free/en/publishers/ceur-workshop-proceedings/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.evalita-1.45/) · [PDF](https://aclanthology.org/2026.evalita-1.45.pdf)

**Citations:** [1](https://md.research.elamur.ai/free/en/papers/munis-2026-malto-svela-specific-attention-head/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses how to check whether machine unlearning in LLMs actually worked. Given an input, the goal is to decide whether the model handled it as retained, forgotten, or never seen. The system was built for the SVELA shared task at EVALITA 2026 and is a white-box membership inference attack that extends MemTrace.

The authors place a trainable soft mask over attention heads and use a LAHIS-style importance score, originally designed to find language-specific heads. This identifies the heads most responsive to each membership class in a single forward and backward pass. Features from these heads are combined with layer-level attention, layer-transition and logit features, and an MLP classifier predicts membership.

Experiments use the synthetic multilingual FAME dataset and Llama-3 models of two sizes, unlearned with five techniques, at both entity and instance level. Macro F1 stays modest, at roughly 0.32–0.35. Ablations, however, show where forgetting leaves traces inside the model, which gives interpretable insight into how unlearning changes internal representations.

## Key points
- **Data and models:** The task uses FAME, a synthetic dataset of 1,000 identities across five languages, of which English is excluded in the challenge. Each identity has 20 QA pairs, and each task provides 912 training pairs. Llama-3-1B and Llama-3-3B were unlearned with five methods:
  - gradient difference,
  - retain-only fine-tuning,
  - KL minimization,
  - gradient ascent,
  - preference optimization.
- **Results:** Task 1 (entity-level) reached macro F1 of 0.336 on the 1B model and 0.353 on the 3B model. Task 2 (instance-level) scored 0.323 for both model sizes.
- **Effect of head selection:** How many heads are selected changes performance depending on task, model size and classifier. Head features help for attribute-level forgetting but contribute little for whole-entity removal.
- **Ablation findings:** In Task 1, transition features carry the most signal, and logits also matter in the larger model. In Task 2, the signal is more evenly spread across feature groups.
- **Unlearning method matters:** Under preference optimization, the forget class consistently scored higher F1 than the unseen class in identity-level classification. Code is available at github.com/MAL-TO/SVELA-2026.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Evren A. Munis, Mattia Sabato, Erfan Bayat and Andrea Lolli. 2026. MALTO at SVELA: A Specific-Attention-Head Approach for Membership Inference Attacks in LLMs Unlearning Evaluation. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{munis2026malto,
  title = {{MALTO at SVELA: A Specific-Attention-Head Approach for Membership Inference Attacks in LLMs Unlearning Evaluation}},
  author = {Evren A. Munis, Mattia Sabato, Erfan Bayat and Andrea Lolli},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.45/}
}
```

- [← previous MALTO at FadeIT: A BERT-Based System for Multi-Label Fallacy Detection in Italian Social Media](https://md.research.elamur.ai/free/en/papers/salami-2026-malto-fadeit-bert-based-system/)
- [next → Mapudungun-Spanish Speech Translation: A Low-Resource End-to-End System for the IWSLT 2026 Shared Task](https://md.research.elamur.ai/free/en/papers/martinez-2026-mapudungun-spanish-speech-translation-low/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/14/)

## References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, Y. Zhang, [A survey on large language model (llm) security and privacy: The good, the bad, and the ugly](https://md.research.elamur.ai/free/en/papers/yao-2023-survey-large-language-model-llm/), High-Confidence Computing 4 (2024) 100211. URL: http://dx.doi.org/10.1016/j.hcc.2024.100211. doi: 10.1016/j.hcc.2024.100211 . [↗](http://dx.doi.org/10.1016/j.hcc.2024.100211)
2. X. Liu, T. Sun, T. Xu, F. Wu, C. Wang, X. Wang, J. Gao, Shield: Evaluation and defense strategies for copyright compliance in llm text generation, 2024. URL: https://arxiv.org/abs/2406.12975. arXiv:2406.12975 . [↗](https://arxiv.org/abs/2406.12975)
3. P. Maini, Z. Feng, A. Schwarzschild, Z. C. Lipton, J. Z. Kolter, [Tofu: A task of fictitious unlearning for llms](https://md.research.elamur.ai/free/en/papers/maini-2024-tofu-task-fictitious-unlearning-llms/), 2024. URL: https://arxiv.org/abs/2401.06121. arXiv:2401.06121 . [↗](https://arxiv.org/abs/2401.06121)
4. C. Savelli, E. Munis, E. Bayat, A. Grieco, F. Giobergia, MALTO at SemEval-2025 task 4: Dual teachers for unlearning sensitive content in LLMs, in: S. Rosenthal, A. Rosá, D. Ghosh, M. Zampieri (Eds.), Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Association fo… [find](https://scholar.google.com/scholar?q=C%2E+Savelli%2C+E%2E+Munis%2C+E%2E+Bayat%2C+A%2E+Grieco%2C+F%2E+Giobergia%2C+MALTO+at+SemEval%2D2025+task+4%3A+Dual+teachers+for+unlearning+sensitive+content+in+LLMs%2C+in%3A+S%2E+Rosenthal%2C+A%2E+Ros%C3%A1%2C+D%2E+Ghosh%2C+M%2E+Zampieri+%28Eds%2E%29%2C+)
5. A. Koudounas, C. Savelli, F. Giobergia, E. Baralis, “Alexa, can you forget me?” Machine Unlearning Benchmark in Spoken Language Understanding, in: Interspeech 2025, 2025, pp. 1768–1772. doi: 10.21437/Interspeech.2025-2607 . — [doi:10.21437/interspeech.2025-2607](https://doi.org/10.21437/interspeech.2025-2607) [find](https://scholar.google.com/scholar?q=A%2E+Koudounas%2C+C%2E+Savelli%2C+F%2E+Giobergia%2C+E%2E+Baralis%2C+%E2%80%9CAlexa%2C+can+you+forget+me%3F%E2%80%9D+Machine+Unlearning+Benchmark+in+Spoken+Language+Understanding%2C+in%3A+Interspeech+2025%2C+2025%2C+pp%2E+1768%E2%80%931772%2E+doi%3A+10%2E21437%2F)
6. C. Savelli, M. La Quatra, A. Koudounas, F. Giobergia, [Svela at evalita 2026: Overview of the selective verification of erasure from llm answers task](https://md.research.elamur.ai/free/en/papers/savelli-2026-svela-evalita-2026-overview-selective/), in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org,…
7. F. Cutugno, A. Miaschi, A. P. Aprosio, G. Rambelli, L. Siciliani, M. A. Stranisci, Evalita 2026: Overview of the 9th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for I… [find](https://scholar.google.com/scholar?q=F%2E+Cutugno%2C+A%2E+Miaschi%2C+A%2E+P%2E+Aprosio%2C+G%2E+Rambelli%2C+L%2E+Siciliani%2C+M%2E+A%2E+Stranisci%2C+Evalita+2026%3A+Overview+of+the+9th+evaluation+campaign+of+natural+language+processing+and+speech+tools+for+italian%2C+in)
8. X. Liu, Q. Song, Q. Zhou, H. Du, S. Xu, W. Jiang, W. Zhang, X. Jia, Focusing on language: Revealing and exploiting language attention heads in multilingual large language models, 2025. URL: https://arxiv.org/abs/2511.07498. arXiv:2511.07498 . — [arXiv:2511.07498](https://arxiv.org/abs/2511.07498) [↗](https://arxiv.org/abs/2511.07498)
9. R. Shokri, M. Stronati, C. Song, V. Shmatikov, [Membership inference attacks against machine learning models](https://md.research.elamur.ai/free/en/papers/shokri-2016-membership-inference-attacks-against-machine/), 2017. URL: https://arxiv.org/abs/1610.05820. arXiv:1610.05820 . [↗](https://arxiv.org/abs/1610.05820)
10. J. Mattern, F. Mireshghallah, Z. Jin, B. Schölkopf, M. Sachan, T. Berg-Kirkpatrick, [Membership inference attacks against language models via neighbourhood comparison](https://md.research.elamur.ai/free/en/papers/mattern-2023-membership-inference-attacks-against-language/), in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023, Association fo…
11. W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, L. Zettlemoyer, Detecting pretraining data from large language models, 2024. URL: https://arxiv.org/abs/2310.16789. arXiv:2310.16789 . [↗](https://arxiv.org/abs/2310.16789)
12. W. Fu, H. Wang, C. Gao, G. Liu, Y. Li, T. Jiang, Mia-tuner: Adapting large language models as pre-training text detector, 2024. URL: https://arxiv.org/abs/2408.08661. arXiv:2408.08661 . — [arXiv:2408.08661](https://arxiv.org/abs/2408.08661) [↗](https://arxiv.org/abs/2408.08661)
13. D. Makhija, M. G. Arivazhagan, V. B. Kumar, R. Gangadharaiah, [Neural breadcrumbs: Membership inference attacks on llms through hidden state and attention pattern analysis](https://md.research.elamur.ai/free/en/papers/makhija-2026-neural-breadcrumbs-membership-inference-attacks/), 2025. URL: https://arxiv.org/abs/2509.05449. arXiv:2509.05449 . [↗](https://arxiv.org/abs/2509.05449)
14. A. Ramakrishna, Y. Wan, X. Jin, K.-W. Chang, Z. Bu, B. Vinzamuri, V. Cevher, M. Hong, R. Gupta, Semeval-2025 task 4: Unlearning sensitive content from large language models, 2025. URL: https://arxiv.org/abs/2504.02883. arXiv:2504.02883 . [↗](https://arxiv.org/abs/2504.02883)
15. M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, H. Hajishirzi, [Do membership inference attacks work on large language models?](https://md.research.elamur.ai/free/en/papers/duan-2024-membership-inference-attacks-work-large/), 2024. URL: https://arxiv.org/abs/2402.07841. arXiv:2402.07841 . [↗](https://arxiv.org/abs/2402.07841)
16. A. Deepak, M. Mou, J. Huang, D. Yang, Identifying unlearned data in LLMs via membership inference attacks, in: C. Christodoulopoulos, T. Chakraborty, C. Rose, V. Peng (Eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=A%2E+Deepak%2C+M%2E+Mou%2C+J%2E+Huang%2C+D%2E+Yang%2C+Identifying+unlearned+data+in+LLMs+via+membership+inference+attacks%2C+in%3A+C%2E+Christodoulopoulos%2C+T%2E+Chakraborty%2C+C%2E+Rose%2C+V%2E+Peng+%28Eds%2E%29%2C+Proceedings+of+the+2025+C)
17. C. Savelli, M. La Quatra, A. Koudounas, F. Giobergia, [FAME: Fictional actors for multilingual erasure](https://md.research.elamur.ai/free/en/papers/savelli-2026-fame-fictional-actors-multilingual-erasure/), in: Proceedings of the Fifteenth Language Resources and Evaluation Conference, European Language Resources Association, 2026.
18. A. Grattafiori, et al., [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/), 2024. URL: https://arxiv.org/abs/2407.21783. arXiv:2407.21783 . [↗](https://arxiv.org/abs/2407.21783)
19. D. Choi, D. Na, Towards machine unlearning benchmarks: Forgetting the personal identities in facial recognition systems, 2023. URL: https://arxiv.org/abs/2311.02240. arXiv:2311.02240 . — [arXiv:2311.02240](https://arxiv.org/abs/2311.02240) [↗](https://arxiv.org/abs/2311.02240)
20. A. Golatkar, A. Achille, S. Soatto, Eternal sunshine of the spotless net: Selective forgetting in deep networks, 2020. URL: https://arxiv.org/abs/1911.04933. arXiv:1911.04933 . — [arXiv:1911.04933](https://arxiv.org/abs/1911.04933) [↗](https://arxiv.org/abs/1911.04933)
21. E. Voita, D. Talbot, F. Moiseev, R. Sennrich, I. Titov, [Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/), in: A. Korhonen, D. Traum, L. Màrquez (Eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Associat…
22. L. Basile, V. Maiorca, D. Doimo, F. Locatello, A. Cazzaniga, Head pursuit: Probing attention specialization in multimodal transformers, 2025. URL: https://arxiv.org/abs/2510.21518. arXiv:2510.21518 . — [arXiv:2510.21518](https://arxiv.org/abs/2510.21518) [↗](https://arxiv.org/abs/2510.21518)
23. P. Guo, A. Syed, A. Sheshadri, A. Ewart, G. K. Dziugaite, Mechanistic unlearning: Robust knowledge unlearning and editing via mechanistic localization, 2024. URL: https://arxiv.org/abs/2410.12949. arXiv:2410.12949 . — [arXiv:2410.12949](https://arxiv.org/abs/2410.12949) [↗](https://arxiv.org/abs/2410.12949)
24. D. P. Kingma, J. Ba, [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/), 2017. URL: https://arxiv.org/abs/ 1412.6980. arXiv:1412.6980 . [↗](https://arxiv.org/abs/)

*Updated 31 October 2026*