---
title: "Beyond the Safety Tax — paper by Meng et al. (2026)"
description: "Most safety defenses for text-to-image diffusion models change the generator itself, either by fine-tuning harmful concepts… 21 references."
authors: ["Xiangtao Meng", "Yingkai Dong", "Ning Yu", "Li Wang", "Zheng Li", "Shanqing Guo"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1394/"
doi: "10.18653/v1/2026.findings-acl.1394"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/meng-2026-beyond-safety-tax-mitigating-unsafe/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/meng-2026-beyond-safety-tax-mitigating-unsafe/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) / Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectification

# Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectification

**Authors:** Xiangtao Meng, Yingkai Dong, [Ning Yu](https://md.research.elamur.ai/free/en/authors/ning-yu/), [Li Wang](https://md.research.elamur.ai/free/en/authors/li-wang/), [Zheng Li](https://md.research.elamur.ai/free/en/authors/zheng-li/) and Shanqing Guo · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1394/) · [PDF](https://aclanthology.org/2026.findings-acl.1394.pdf) · [DOI 10.18653/v1/2026.findings-acl.1394](https://doi.org/10.18653/v1/2026.findings-acl.1394) · [OpenAlex](https://openalex.org/W4415986797)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Most safety defenses for text-to-image diffusion models change the generator itself, either by fine-tuning harmful concepts away (e.g., ESD, SafeGEN) or by steering sampling at inference (e.g., SLD). Unsafe and benign concepts are entangled in the latent space, so these edits also harm harmless content. The authors call this quality loss the "Safety Tax." Their alternative, SafePatch, is a separate safety module attached to a frozen base model. Its backbone is a trainable copy of the base model's encoder. It joins the generator through zero-initialized convolutions, so it has no effect when training begins. An instruction-aware spatial projection helps it pinpoint where unsafe content appears. Training relies on ACS, a new counterfactual dataset. An LLM minimally rewrites unsafe Lexica prompts into safe versions, and the target model renders safe images. A three-stage filter (automated auditors, semantic check, human review) then keeps only tightly aligned pairs. Against six defenses, SafePatch suppresses unsafe outputs more strongly, including under adversarial prompts, while preserving image quality and text alignment. This suggests safety can be added without sacrificing benign fidelity.

## Key points
- On I2P, SafePatch lowers the overall unsafe probability to 7%, while the baselines stay around 20%.
- Under adversarial attacks, manual checks found 0% unsafe images for SneakyPrompt. Ring-A-Bell yielded 24% unsafe for violence prompts and 6% for nudity prompts, better than every baseline.
- Scaling ACS from 1K to 10K pairs improves final defense performance. It also halves the iterations needed to reach near-saturation, from 50K to 25K.
- On one RTX 5090 with SD v1.4, the module adds about 1122 ms of latency per generation and 766 MB of GPU memory.
- Unsafe rates are measured with NudeNet and Q16, which the authors acknowledge are imperfect proxies. The ACS prompts are shared only on request for research use.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shandong University](https://md.research.elamur.ai/free/en/universities/shandong-university/) — University, Jinan, China · 19 works · 952 citations

Also on the byline: [Netflix](https://ror.org/0197qw696).

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Xiangtao Meng, Yingkai Dong, Ning Yu, Li Wang, Zheng Li and Shanqing Guo. 2026. Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectification. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{meng2026beyond,
  title = {{Beyond the Safety Tax: Mitigating Unsafe Text-to-Image Generation via External Safety Rectification}},
  author = {Xiangtao Meng, Yingkai Dong, Ning Yu, Li Wang, Zheng Li and Shanqing Guo},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1394},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1394}
}
```

- [← previous Beyond the Panorama: Training-Free Hierarchical Perception-Reasoning for Fine-Grained Vision in MLLMs](https://md.research.elamur.ai/free/en/papers/yi-2026-beyond-panorama-training-free-hierarchical/)
- [next → Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation](https://md.research.elamur.ai/free/en/papers/zhang-2026-beyond-singular-revealing-value-multiple/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/)

## References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2023. Nsfw gpt. [https://www.reddit.com/r/](https://www.reddit.com/r/) ChatGPT/comments/11vlp7j/nsfwgpt_that_ nsfw_prompt/. 2023. Usage policies of openai. https://openai. com/policies/usage-policies/. [↗](https://www.reddit.com/r/)
2. Ibtihel Amara, Ahmed Imtiaz Humayun, Ivana Kajic, Zarana Parekh, Natalie Harris, Sarah Young, Chirag Nagpal, Najoung Kim, Junfeng He, Cristina Nader Vasconcelos, and 1 others. 2025. Erasing more than intended? how concept erasure degrades the generation of non-target concepts. In Proceedings of the… [find](https://scholar.google.com/scholar?q=Ibtihel+Amara%2C+Ahmed+Imtiaz+Humayun%2C+Ivana+Kajic%2C+Zarana+Parekh%2C+Natalie+Harris%2C+Sarah+Young%2C+Chirag+Nagpal%2C+Najoung+Kim%2C+Junfeng+He%2C+Cristina+Nader+Vasconcelos%2C+and+1+others%2E+2025%2E+Erasing+more+than+)
3. P Bedapudi. 2019. Nudenet: Neural nets for nudity classification, detection and selective censoring. [find](https://scholar.google.com/scholar?q=P+Bedapudi%2E+2019%2E+Nudenet%3A+Neural+nets+for+nudity+classification%2C+detection+and+selective+censoring%2E)
4. Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. 2021. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963. — [arXiv:2110.01963](https://arxiv.org/abs/2110.01963) [find](https://scholar.google.com/scholar?q=Abeba+Birhane%2C+Vinay+Uday+Prabhu%2C+and+Emmanuel+Kahembwe%2E+2021%2E+Multimodal+datasets%3A+misogyny%2C+pornography%2C+and+malignant+stereotypes%2E+arXiv+preprint+arXiv%3A2110%2E01963%2E+%E2%80%94+%5BarXiv%3A2110%2E01963%5D%28https%3A%2F%2Farxi)
5. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+and+1+others%2E+2022%2E+Toy+models+of+superpositi)
6. Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. 2023. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2426–2436. [find](https://scholar.google.com/scholar?q=Rohit+Gandikota%2C+Joanna+Materzynska%2C+Jaden+Fiotto%2DKaufman%2C+and+David+Bau%2E+2023%2E+Erasing+concepts+from+diffusion+models%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision%2C+page)
7. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86– 92. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+Iii%2C+and+Kate+Crawford%2E+2021%2E+Datasheets+for+datasets%2E+Communications+of+the+ACM%2C+64%2812%29%3A86%E2%80%93+92%2E)
8. Gabriel Goh, James Betker, Li Jing, and Aditya Ramesh. 2024. Dall·e 3. [https://openai.com/index/](https://openai.com/index/) dall-e-3/. [↗](https://openai.com/index/)
9. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. [Clipscore: A reference-free evaluation metric for image captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/). arXiv preprint arXiv:2104.08718.
10. Byung Hyun Lee, Sungjin Lim, and Se Young Chun. 2025. Localized concept erasure for text-to-image diffusion models using training-free gated low-rank adaptation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18596– 18606. [find](https://scholar.google.com/scholar?q=Byung+Hyun+Lee%2C+Sungjin+Lim%2C+and+Se+Young+Chun%2E+2025%2E+Localized+concept+erasure+for+text%2Dto%2Dimage+diffusion+models+using+training%2Dfree+gated+low%2Drank+adaptation%2E+In+Proceedings+of+the+Computer+Vision+)
11. Hang Li, Chengzhi Shen, Philip Torr, Volker Tresp, and Jindong Gu. 2024a. Self-discovering interpretable diffusion latent directions for responsible text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Hang+Li%2C+Chengzhi+Shen%2C+Philip+Torr%2C+Volker+Tresp%2C+and+Jindong+Gu%2E+2024a%2E+Self%2Ddiscovering+interpretable+diffusion+latent+directions+for+responsible+text%2Dto%2Dimage+generation%2E+In+Proceedings+of+the+IEE)
12. Xinfeng Li, Yuchen Yang, Jiangyi Deng, Chen Yan, Yanjiao Chen, Xiaoyu Ji, and Wenyuan Xu. 2024b. Safegen: Mitigating sexually explicit content generation in text-to-image models. In arXiv preprint arXiv:2404.06666. — [arXiv:2404.06666](https://arxiv.org/abs/2404.06666) [find](https://scholar.google.com/scholar?q=Xinfeng+Li%2C+Yuchen+Yang%2C+Jiangyi+Deng%2C+Chen+Yan%2C+Yanjiao+Chen%2C+Xiaoyu+Ji%2C+and+Wenyuan+Xu%2E+2024b%2E+Safegen%3A+Mitigating+sexually+explicit+content+generation+in+text%2Dto%2Dimage+models%2E+In+arXiv+preprint+arX)
13. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pag…
14. Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. 2024. Latent guard: a safety framework for text-to-image generation. In European Conference on Computer Vision, pages 93– 109. Springer. [find](https://scholar.google.com/scholar?q=Runtao+Liu%2C+Ashkan+Khakzar%2C+Jindong+Gu%2C+Qifeng+Chen%2C+Philip+Torr%2C+and+Fabio+Pizzati%2E+2024%2E+Latent+guard%3A+a+safety+framework+for+text%2Dto%2Dimage+generation%2E+In+European+Conference+on+Computer+Vision%2C+pag)
15. Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. 2024. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6430–6440. [find](https://scholar.google.com/scholar?q=Shilin+Lu%2C+Zilan+Wang%2C+Leyang+Li%2C+Yanzhu+Liu%2C+and+Adams+Wai%2DKin+Kong%2E+2024%2E+Mace%3A+Mass+concept+erasure+in+diffusion+models%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Rec)
16. ML Research. 2023. Q16: A multi-category safety classifier. [https://github.com/ml-research/Q16](https://github.com/ml-research/Q16). Accessed: 2024. notAI Tech. 2019. Nudenet: Neural nets for nudity detection. https://github.com/notAI-tech/ [↗](https://github.com/ml-research/Q16)
17. NudeNet. Accessed: 2024. [find](https://scholar.google.com/scholar?q=NudeNet%2E+Accessed%3A+2024%2E)
18. Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. 2022. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11410–11420. [find](https://scholar.google.com/scholar?q=Gaurav+Parmar%2C+Richard+Zhang%2C+and+Jun%2DYan+Zhu%2E+2022%2E+On+aliased+resizing+and+surprising+subtleties+in+gan+evaluation%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recogniti)
19. Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramèr. 2022. Red-teaming the stable diffusion safety filter. ArXiv, abs/2210.04610. — [arXiv:2210.04610](https://arxiv.org/abs/2210.04610) [find](https://scholar.google.com/scholar?q=Javier+Rando%2C+Daniel+Paleka%2C+David+Lindner%2C+Lennard+Heim%2C+and+Florian+Tram%C3%A8r%2E+2022%2E+Red%2Dteaming+the+stable+diffusion+safety+filter%2E+ArXiv%2C+abs%2F2210%2E04610%2E+%E2%80%94+%5BarXiv%3A2210%2E04610%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F22)
20. Robin Rombach, Patrick Esser. 2022. Stable diffusion v1-4. [https://huggingface.co/CompVis/](https://huggingface.co/CompVis/) stable-diffusion-v1-4. [↗](https://huggingface.co/CompVis/)
21. Robin Rombach, Patrick Esser. 2023a. Stable diffusion safety checker. [https://huggingface.co/](https://huggingface.co/) CompVis/stable-diffusion-safety-checker. [↗](https://huggingface.co/)

*Updated 1 November 2026*