---
title: "Selective Span-Level Unlearning for Large Language Models"
description: "The authors propose a selective span-level method that uses only signals from the model itself. 20 references."
authors: ["Chaewon Yoon", "Dongjun Kim", "Hyun-Je Song"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.acl-short.35/"
doi: "10.18653/v1/2026.acl-short.35"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yoon-2026-selective-span-level-unlearning-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yoon-2026-selective-span-level-unlearning-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/49/) / Selective Span-Level Unlearning for Large Language Models

# Selective Span-Level Unlearning for Large Language Models

**Authors:** Chaewon Yoon, Dongjun Kim and Hyun-Je Song · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-short.35/) · [PDF](https://aclanthology.org/2026.acl-short.35.pdf) · [DOI 10.18653/v1/2026.acl-short.35](https://doi.org/10.18653/v1/2026.acl-short.35) · [OpenAlex](https://openalex.org/W7166887169)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Unlearning in LLMs tries to erase memorized private or copyrighted material without retraining from scratch. Applying the forget loss to whole sequences also wipes out harmless linguistic content and degrades utility. Earlier selective methods narrow the target, but they usually rely on auxiliary models or annotations to decide what to forget, which may not match what the target model actually relies on. The authors propose a selective span-level method that uses only signals from the model itself. First, each token gets an importance score by comparing its gradient against the difference between the mean forget-set and retain-set gradients, scaled by an approximate inverse Hessian. High-scoring tokens become anchors. The model then repeatedly samples spans around each anchor, and only tokens that appear consistently are kept as forget targets. These spans define a binary-weighted forget loss that can be plugged into GA, NPO or SO-NPO. On TOFU and MUSE-News, the method forgets about as well as prior selective approaches while retaining considerably more knowledge. This shows that precise unlearning can work without external supervision.

## Key points
- **Benchmarks:** Evaluation uses TOFU (standard "forget10" split: 10% forget, 90% retain) and MUSE-News (official entity-based forget setting).
- **Baselines and backbones:** The method is compared with the selective baselines SU, SEUL and WTNPO. It is applied on top of GA, NPO and SO-NPO, including variants with KL-divergence retention (e.g., SPAN-SO-NPO+KL).
- **MUSE-News results:** Prior selective methods drove VerbMem and KnowMem close to zero even on the retain set, which indicates over-forgetting. The proposed variants stayed close to the retrained reference model.
- **Self-consistency ablation (TOFU):** Span selection via self-consistency beat plain token-level selection on both model utility and forget quality.
- **Weighting ablation (TOFU):** Binary weights produced better forget quality than soft frequency-based weights:
  - SPAN-NPO: −5.11 vs −11.85
  - SPAN-SO-NPO: −8.83 vs −11.05

  The stated limitation is the extra compute needed for multiple generations per anchor token.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Jeonbuk National University](https://md.research.elamur.ai/free/en/universities/jeonbuk-national-university/) — University, Jeonju, South Korea · 3 works · 205 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Chaewon Yoon, Dongjun Kim and Hyun-Je Song. 2026. Selective Span-Level Unlearning for Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yoon2026selective,
  title = {{Selective Span-Level Unlearning for Large Language Models}},
  author = {Chaewon Yoon, Dongjun Kim and Hyun-Je Song},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.35},
  url = {https://doi.org/10.18653/v1/2026.acl-short.35}
}
```

- [← previous Selective Multimodal Retrieval for Automated Verification of Image–Text Claims](https://md.research.elamur.ai/free/en/papers/tsoneva-2026-selective-multimodal-retrieval-automated-verification/)
- [next → Self-Consistency from Only Two Samples: CoT–PoT Ensembling for Efficient LLM Reasoning](https://md.research.elamur.ai/free/en/papers/saparkhan-2026-self-consistency-only-two-samples/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/49/)

## References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zachary Coalson, Juhan Bae, Nicholas Carlini, and Sanghyun Hong. 2025. IF-guide: Influence function-guided detoxification of LLMs. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Zachary+Coalson%2C+Juhan+Bae%2C+Nicholas+Carlini%2C+and+Sanghyun+Hong%2E+2025%2E+IF%2Dguide%3A+Influence+function%2Dguided+detoxification+of+LLMs%2E+In+The+Thirty%2Dninth+Annual+Conference+on+Neural+Information+Processin)
2. Ronen Eldan and Mark Russinovich. 2024. Who’s harry potter? approximate unlearning for LLMs. [find](https://scholar.google.com/scholar?q=Ronen+Eldan+and+Mark+Russinovich%2E+2024%2E+Who%E2%80%99s+harry+potter%3F+approximate+unlearning+for+LLMs%2E)
3. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive mfultitask language understanding. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Steven+Basart%2C+Andy+Zou%2C+Mantas+Mazeika%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+2021%2E+Measuring+massive+mfultitask+language+understanding%2E+In+International+Conference+on+Learnin)
4. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. [Knowledge unlearning for mitigating privacy risks in language models](https://md.research.elamur.ai/free/en/papers/jang-2022-knowledge-unlearning-mitigating-privacy-risks/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
5. Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024. [SOUL: Unlocking the power of second-order optimization for LLM unlearning](https://md.research.elamur.ai/free/en/papers/jia-2024-soul-unlocking-power-second-order/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages…
6. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. [Copyright violations and large language models](https://md.research.elamur.ai/free/en/papers/karamolegkou-2023-copyright-violations-large-language-models/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7403–7412, Singapore. Association for Computational Linguistics.
7. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. 2024. [TOFU: A task of fictitious unlearning for LLMs](https://md.research.elamur.ai/free/en/papers/maini-2024-tofu-task-fictitious-unlearning-llms/). In First Conference on Language Modeling.
8. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
9. Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2024. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. In Neurips Safe Generative AI Workshop 2024. [find](https://scholar.google.com/scholar?q=Chongyu+Fan%2C+Jiancheng+Liu%2C+Licong+Lin%2C+Jinghan+Jia%2C+Ruiqi+Zhang%2C+Song+Mei%2C+and+Sijia+Liu%2E+2024%2E+Simplicity+prevails%3A+Rethinking+negative+preference+optimization+for+llm+unlearning%2E+In+Neurips+Safe+Ge)
10. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. 2025. MUSE: Machine unlearning six-way evaluation for language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Jaechan+Lee%2C+Yangsibo+Huang%2C+Sadhika+Malladi%2C+Jieyu+Zhao%2C+Ari+Holtzman%2C+Daogao+Liu%2C+Luke+Zettlemoyer%2C+Noah+A%2E+Smith%2C+and+Chiyuan+Zhang%2E+2025%2E+MUSE%3A+Machine+unlearning+six%2Dway+evaluation+fo)
11. XiaoHua Feng, Chaochao Chen, Yuyuan Li, and Zibin Lin. 2024. Fine-grained pluggable gradient ascent for knowledge unlearning in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10141– 10155, Miami, Florida, USA. Association for Computa… [find](https://scholar.google.com/scholar?q=XiaoHua+Feng%2C+Chaochao+Chen%2C+Yuyuan+Li%2C+and+Zibin+Lin%2E+2024%2E+Fine%2Dgrained+pluggable+gradient+ascent+for+knowledge+unlearning+in+language+models%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Meth)
12. Yixin Wan, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, and Rahul Gupta. 2025. Not every token needs forgetting: Selective unlearning balancing forgetting and utility in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 1827–1835, Suzhou, China… [find](https://scholar.google.com/scholar?q=Yixin+Wan%2C+Anil+Ramakrishna%2C+Kai%2DWei+Chang%2C+Volkan+Cevher%2C+and+Rahul+Gupta%2E+2025%2E+Not+every+token+needs+forgetting%3A+Selective+unlearning+balancing+forgetting+and+utility+in+large+language+models%2E+In+F)
13. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, and 1 others. 2023. [Studying large language model generalization with influence functions](https://md.research.elamur.ai/free/en/papers/grosse-2023-studying-large-language-model-generalization/). arXiv preprint arXiv:2308.03296.
14. Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. 2025a. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 843–851. [find](https://scholar.google.com/scholar?q=Lingzhi+Wang%2C+Xingshan+Zeng%2C+Jinsong+Guo%2C+Kam%2DFai+Wong%2C+and+Georg+Gottlob%2E+2025a%2E+Selective+forgetting%3A+Advancing+machine+unlearning+techniques+and+evaluation+in+language+models%2E+In+Proceedings+of+the)
15. Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, and Kilian Q Weinberger. 2025b. Rethinking llm unlearning objectives: A gradient perspective and go beyond. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Qizhou+Wang%2C+Jin+Peng+Zhou%2C+Zhanke+Zhou%2C+Saebyeol+Shin%2C+Bo+Han%2C+and+Kilian+Q+Weinberger%2E+2025b%2E+Rethinking+llm+unlearning+objectives%3A+A+gradient+perspective+and+go+beyond%2E+In+International+Conference+)
16. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
17. Note that the inverse Hessian approximation is computed only once per model and can be reused across tokens. Span generation for each anchor token is independent and can be parallelized with sufficient computational resources. Therefore, the proposed method remains practically scalable. *Table 5: Hy… — [Machine Unlearning of Pre-trained Large Language Models](https://md.research.elamur.ai/free/en/papers/yao-2024-machine-unlearning-pre-trained-large/)
18. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024b. [Large language model unlearning](https://md.research.elamur.ai/free/en/papers/yao-2023-large-language-model-unlearning/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
19. Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. 2025. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. AI and Ethics, 5(3):2445–2454. [find](https://scholar.google.com/scholar?q=Dawen+Zhang%2C+Pamela+Finckenberg%2DBroman%2C+Thong+Hoang%2C+Shidong+Pan%2C+Zhenchang+Xing%2C+Mark+Staples%2C+and+Xiwei+Xu%2E+2025%2E+Right+to+be+forgotten+in+the+era+of+large+language+models%3A+Implications%2C+challenges%2C)
20. Batch size SU SEUL WTNPO NPO SimNPO SO-NPO 16 16 16 16 16 16 NPO SimNPO SO-NPO 16 16 16 NPO SimNPO SO-NPO 16 16 16 [find](https://scholar.google.com/scholar?q=Batch+size+SU+SEUL+WTNPO+NPO+SimNPO+SO%2DNPO+16+16+16+16+16+16+NPO+SimNPO+SO%2DNPO+16+16+16+NPO+SimNPO+SO%2DNPO+16+16+16)

*Updated 2 November 2026*