Beautiful Images, Toxic Words: Understanding and Addressing Offensive Text in AI-Generated Images
1CISPA Helmholtz Center for Information Security
2University of Toronto
3Vector Institute
*Indicates Equal Contribution
To reduce accidental exposure, all such images are blurred by default.
Hover over an image to reveal it.
Text-to-image diffusion models have become remarkably good at generating images. With the right prompt, these models can generate photorealistic scenes, stylized illustrations, and even readable text rendered into signs, storefronts, posters, or clothing. But with this new capability comes a risk that has received surprisingly little attention. We are the first to uncover a new safety threat: state-of-the-art text-to-image models can embed NSFW language directly within the images they generate.
An Overlooked Failure Mode in Image Safety
In our work, we found that state-of-the-art models, including SD3, SDXL, FLUX, and DeepFloyd IF, are vulnerable to producing harmful embedded text. This includes slurs, personal insults, and sexually explicit language. The image below shows examples from four different models when prompted with NSFW phrases.
Why Existing Defenses Fail
Naive safety solutions, such as prompt filters and OCR-based image detectors, often fail to catch this issue. Filters can miss subtle or context-dependent cases, and OCR-based systems are easy to bypass when the text is stylized or distorted, as shown below:
Recent work has explored mitigation strategies for diffusion models, including filtering unsafe embeddings during training or inference: SafeCLIP, suppressing neurons statistically linked to harmful generations: AURA, or editing diffusion steps to steer outputs: ESD. However, none of these approaches achieve a good trade-off between reducing harmful content and preserving the model’s ability to render benign, legible text. In practice, they often degrade output quality or remove safe generations along with the unsafe ones as we show in our paper.
A Targeted Intervention
We propose a targeted fine-tuning method that focuses only on the model components responsible for generating text in the image. Instead of retraining the full model or modifying the text encoder, we fine-tune a small subset of layers. These layers are selected based on prior work identifying where text generation occurs in the diffusion process. For each NSFW prompt, we create a benign alternative word using GPT-4 that is semantically similar to the original NSFW word and matches its token length. We then generate image pairs: one from the original NSFW prompt and one from the modified prompt with the benign word substituted. Both images are rendered using the same diffusion model settings at 1024x1024 resolution, ensuring they differ only in the embedded text (as shown in the figure below). The model is then trained to produce the safer image even when it receives the original, NSFW prompt.
➡️
What makes this approach effective is its precision. Rather than trying to solve the problem broadly, we adjust only what is necessary. The result is that the model learns to suppress harmful terms while continuing to produce sharp, coherent, and legible text when it is safe to do so.
ToxicBench: A Benchmark for Harmful Text in Images
To enable broader progress on this issue, we introduce ToxicBench, a safety benchmark specifically designed to evaluate whether diffusion models generate offensive text in images. It provides a standardized testbed for the community to measure and compare safety interventions. ToxicBench built on CreativeBench includes 437 NSFW terms with corresponding benign substitutes, 218 prompt templates that encourage text rendering, and a test set with unseen NSFW words to evaluate generalization. The benchmark also includes a modular evaluation pipeline (see figure below) that combines OCR, a toxicity classifier, and text similarity metrics.
Central to our evaluation is a novel n-gram Levenshtein metric, which we introduce to assess whether the NSFW term has been removed and the intended benign term rendered instead. This metric compares OCR output to both the original NSFW word and its benign counterpart, providing word-level edit distance measurements. Unlike standard character-level Levenshtein, it captures whether a substitution occurred cleanly, even when only a single word changes and the rest of the image remains untouched.
Using ToxicBench, we evaluated our intervention across three models: SD3, SDXL, and DeepFloyd IF. The method significantly reduced the presence of harmful text while preserving benign generations and overall image quality. Images shown below are from SD3 for the prompt template: “A little frog holding a sign saying [word]”
Conclusion
As generative models continue to make their way into creative tools, search engines, and public interfaces, embedded NSFW text poses serious moderation challenges. It is easy to miss, difficult to detect with standard tools, and potentially harmful to users and platform trust.
Our method provides a practical solution. It avoids changing the prompt or modifying tokenization. Instead, it applies precise updates only where the harmful content arises. To ensure that these targeted changes are both effective and reliable, we evaluated our approach using ToxicBench, our dedicated benchmark for detecting harmful text in images.
Learn More
Join the Effort to Build Safer AI
If you are interested in advancing AI safety, we invite you to contribute to ToxicBench and collaborate with the community to build more ethical and responsible generative models. The future of AI creativity depends on it.