Debiasing CLIP with Feature Augmentation and Selective Update to Preserve Zero-Shot Capability

Vision-language models, including CLIP, have shown impressive performance across various tasks. However, these models often struggle to distinguish meaningful features from spurious correlations in dataset biases, resulting in gen-eralization issues. Mitigating these biases without compromising CLIP's core strengths for zero-shot performance is crucial but has been overlooked. We propose a novel adversarial training method to debias CLIP, combining PGD on the projection layer for feature augmentation and Selective Update for Debiasing (SUD). Our method targets the projection layer, using minimally distorted feature augmentation and selective update strategy guided by model predictions. PGD uses the embedding layer's gradients to generate biased features that induce misclassifications with minimal distortion. SUD applies different objective functions to refine features based on prediction accuracy. These methods effectively mitigate bias by training on misclassified samples and preserve the existing embedding space. Experiments confirm our method improves model robustness against biases while maintaining zero-shot performance. This approach offers a promising solution for debiasing vision-language models without degradation in performance.