Debiasing Classifiers by Amplifying Bias with Latent Diffusion and Large Language Models

Neural networks struggle with image classification when biases are learned and misleads correlations, affecting their generalization and performance. Previous methods require attribute labels (e.g. background, color) or utilizes Generative Adversarial Networks (GANs) to mitigate biases. We introduce DiffuBias, a novel pipeline for text-to-image generation that generates bias-conflict samples, without any training. By utilizing pretrained diffusion and image captioning models, DiffuBias generates, bias-conflict samples using the top-K losses from a biased classifier (f_B) to debias the classifier. This method not only debiases effectively but also boosts classifier generalization capabilities. Our comprehensive experimental evaluations demonstrate that DiffuBias achieves state-of-the-art performance on benchmark datasets.