Fine-Grained Concept Erasure: This figure demonstrates the issue of collateral forgetting (termed as \textit{adjacency}) in selective concept erasure using existing state-of-the-art algorithms in text-to-image diffusion-based foundation models. It highlights the inability of methods that can precisely erase target concepts from a model’s knowledge while preserving its ability to generate closely related concepts.

Abstract

Existing unlearning algorithms in text-to-image generative models often fail to preserve the knowledge of semantically related concepts when removing specific target concepts: a challenge known as adjacency. To address this, we propose FADE (Fine grained Attenuation for Diffusion Erasure), introducing adjacency aware unlearning in diffusion models. FADE comprises two components: (1) the Concept Neighborhood, which identifies an adjacency set of related concepts, and (2) Mesh Modules, employing a structured combination of Expungement, Adjacency, and Guidance loss components. These enable precise erasure of target concepts while preserving fidelity across related and unrelated concepts. Evaluated on datasets like Stanford Dogs, Oxford Flowers, CUB, I2P, Imagenette, and ImageNet1k, FADE effectively removes target concepts with minimal impact on correlated concepts, achieving atleast a 12% improvement in retention performance over state-of-the-art methods.

arXiv preprint

BibTex:

@inproceedings{FADE2025,
  title={Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models},
  author={Kartik Thakral and Tamar Glaser and Tal Hassner and Mayank Vatsa and Richa Singh},
  booktitle={CVF/IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
  URL = {\url{https://talhassner.github.io/home/publication/2025_CVPR}},
  year={2025}
}