Accepted at ICML 2026

Paper  |  Code  |  BibTeX

Modern text-to-image models such as SD3.5, FLUX, and Infinity can generate photorealistic images from text prompts. They are trained on massive datasets collected from the internet. These datasets contain concepts that we may not want the models to reproduce. Examples include unique artistic styles, copyrighted characters, and explicit content.

Removing such concepts after training is difficult. Retraining a model from scratch without the unwanted data is usually impractical since these systems contain billions of parameters and require enormous computational resources. We therefore need a method that can block a target concept without retraining the model.

To address this challenge, we introduce BLOCK (Bottleneck-Layer-Oriented Concept Knockout). Our method leaves almost the entire model unchanged because it targets one narrow point: the transformation layers that connect the text encoder to the image-generating backbone, through which every prompt must pass. BLOCK replaces these layers with an editable module called a transcoder. The transcoder suppresses the target concept while the generator itself remains untouched. Our modification is integrated directly into the model, so it cannot be easily removed or bypassed.

Why existing methods fall short

Existing concept-removal methods fall into two categories, and each has a weakness that BLOCK avoids.

Internal methods modify the model’s weights, which are learned during training. Training-based approaches retrain the model so that it no longer produces the target concept. This process is expensive and requires a curated set of example images. Closed-form approaches avoid retraining and instead apply a direct update to the weights. The size of this update creates a trade-off. Small changes leave traces of the concept, while large changes degrade unrelated generations. Both classes of internal methods also suffer under repeated use because each edit builds on the last and the errors accumulate.

External methods leave the weights untouched and suppress the concept from outside the model. They do this either during generation or through a separate filtering module. The underlying model still retains the concept. Since the filter is detachable, anyone with access to the model can remove it and restore the original behavior. This limitation is especially important for open-weight models whose parameters are publicly available.

BLOCK combines the advantages of both approaches. Like an external method, it is cheap to apply. Like an internal method, it is integrated into the model itself and cannot be detached without modifying the model.

The Bottleneck

To understand where BLOCK intervenes, consider how a prompt moves through the model. First, a text encoder converts the prompt into an embedding, a numerical representation that captures its semantic meaning. This embedding is then used to condition the generative backbone, the large neural network responsible for producing the final image.

Before the embedding can reach the backbone, it passes through one or two narrow transformation layers that map it into the form the backbone expects. We call this the bottleneck layer, because every concept carried by the text embedding passes through it. SD3.5 and FLUX have two such layers, and Infinity has one. BLOCK operates at these layers.

The bottleneck is the ideal place to intervene: if a concept never passes through it, the backbone can never render it. A direct edit is still impossible, because the units inside this layer are polysemantic, meaning each one responds to many unrelated concepts. There is no single unit for “Van Gogh” to disable. The concept is spread across many units that also represent other concepts, so disabling one would damage everything else that depends on it.

Our Intervention

We replace this bottleneck layer with the transcoder: a small neural network that reproduces what the original layer computed but re-expresses the information so that its internal units become monosemantic. We call these units latents. This separation comes from a TopK activation, which keeps only the K most strongly responding latents active for any prompt (K is a small fixed number) and sets the rest to zero. Since only a few latents can be active at once, the transcoder is forced to give each concept its own latents. As a result, “Van Gogh” is represented by a small set of identifiable latents rather than spread across the whole layer.

We build the transcoder without retraining the image generator. Its training data is a set of prompts taken from open source datasets. We forward each prompt through the model’s text encoder to obtain the transcoder’s inputs, and pass those through the original bottleneck layer to obtain the matching outputs. The transcoder is trained on these input-output pairs to reproduce the original layer while keeping its latents sparse. No gradients propagate through the backbone, which keeps the procedure inexpensive. We then insert the trained transcoder in place of the original layer.

Removing a concept then requires no further training. We identify the latents that activate for the concept’s tokens (for “Van Gogh”, the latents for “Van” and “Gogh”) and redirect them to reproduce the output of the empty token, the input that represents a prompt with no text. The concept is intercepted at the bottleneck and never reaches the backbone, while every other concept is unaffected. This change is to weights inside the model, so it cannot be detached like an external filter, and it survives even if the text encoder in front of it is replaced.

The figure below contrasts the two cases. In the original model (top), the bottleneck passes the concept through to the backbone. With BLOCK (bottom), the transcoder redirects the concept to the empty token, so the backbone never receives it.

Results

We test BLOCK on UnlearnCanvas, a standard benchmark for measuring how well a method removes a concept while leaving everything else intact. We evaluate it on two Diffusion Models (SD3.5 and FLUX) and two Image AutoRegressive models (Infinity-2B and Infinity-8B), covering both dominant model families. The benchmark targets two kinds of concepts: an artistic style (such as Pop Art or Cubism) and an object (such as a butterfly).

We compare BLOCK against two main baselines, UCE and LOCOEDIT. We chose these two because they are the most relevant internal methods that, like BLOCK, apply to both Diffusion Models and Image AutoRegressive models. UCE was the strongest prior method for the removal task itself, and LOCOEDIT makes a small localized edit that preserves the rest of the model. In the paper we additionally compare against more baselines, such as EraseAnything and SAeUron. BLOCK outperforms all of them.

We evaluate with three standard metrics. Unlearning accuracy (UA) is how often the target concept is successfully removed. In-domain retain accuracy (IRA) is how well the model keeps other concepts of the same kind, for example other objects when removing an object. Cross-domain retain accuracy (CRA) is how well it keeps concepts of the other kind, for example styles when removing an object.

Across both model families and both kinds of target, BLOCK removes the concept more thoroughly than these baselines while better preserving unrelated content. On SD3.5, for example, it improves style UA by about 8 percentage points over the best baseline.

Style removal

Object removal

In practice we rarely remove just one concept. A useful method must support sequential removal, where concepts are removed one at a time, without degrading the model. This is where the internal-method baselines struggle most. Each removal is a separate edit to the model’s weights, and every edit introduces a small error. These errors accumulate as more concepts are removed, progressively degrading the model.

BLOCK does not have this problem. Each concept is represented by its own separate set of transcoder features, so removing ten concepts amounts to redirecting ten independent sets of features, with no interference between them. The outcome is identical whether the concepts are removed together or one at a time. The plots below show the result: BLOCK remains stable as more concepts are removed, while the baselines decline.

A removed concept is only truly gone if it cannot be recovered through clever prompting. Adversarial prompting is the practice of deliberately constructing unusual prompts that cause a model to produce content the removal was meant to prevent. A determined user will not simply type “Van Gogh”. Instead, they will search for alternative wordings that evade the removal, and a removal method is only trustworthy if it remains robust.

We test this with Ring-A-Bell, an established attack that automatically searches for prompts which reintroduce a suppressed concept. We chose it because it is a widely used, model-agnostic attack designed specifically to evaluate concept-removal methods. Ring-A-Bell operates in a black-box setting: it searches only over prompts and does not access the model’s internals. The table below reports the attack success rate, the percentage of adversarial prompts that succeed in recovering the concept. BLOCK achieves the lowest attack success rate on every model.

Finally, we can see the effect directly: in each pair below, BLOCK removes the targeted concept while the rest of the image is preserved.

Conclusion

As image-generation models become more capable and more widely deployed, the ability to reliably remove specific concepts is becoming a practical requirement rather than a research curiosity. The existing approaches each fall short in a specific way. Internal methods modify the model broadly and degrade as more concepts are removed. External methods leave the model intact and can be detached. BLOCK avoids both failures at once. It turns a single bottleneck layer into a precise and permanent mechanism for removing concepts that works across both model families.

We hope BLOCK offers a practical foundation for safer and more controllable deployment of frontier image-generation models.

Learn more

• Read the full paper: Arxiv
• Explore the code: GitHub

BibTeX

@inproceedings{kumar2026concept,
  title = {Concept Removal for Frontier Image Generative Models},
  author = {Kumar, Aditya and Joly, Pierre and Dziedzic, Adam and Boenisch, Franziska},
  booktitle = {Forty-third International Conference on Machine Learning (ICML)},
  year = {2026}
}