Abstract
We propose a novel approach to mitigate biases in computer vision models by utilizing counterfactual generation and fine-tuning. While counterfactuals have been used to analyze and address biases in DNN models, the counterfactuals themselves are often generated from biased generative models, which can introduce additional biases or spurious correlations. To address this issue, we propose using adversarial images, that is images that deceive a deep neural network but not humans, as counterfactuals for fair model training. Our approach leverages a curriculum learning framework combined with a fine-grained adversarial loss to fine-tune the model using adversarial examples. By incorporating adversarial images into the training data, we aim to prevent biases from propagating through the pipeline. We validate our approach through both qualitative and quantitative assessments, demonstrating improved bias mitigation and accuracy compared to existing methods. Qualitatively, our results indicate that post-training, the decisions made by the model are less dependent on the sensitive attribute and our model better disentangles the relationship between sensitive attributes and classification variables.
Abstract (translated)
我们提出了一种通过利用反事实生成和微调来减轻计算机视觉模型中偏见的新方法。虽然反事实已经被用于分析并解决DNN模型的偏见，但反事实本身通常是从有偏的生成模型中生成的，这可能会引入额外的偏见或伪相关关系。为了解决这个问题，我们提出使用对抗性图像作为公平模型训练的反事实。我们的方法结合了级联学习框架和细粒度对抗损失，通过对抗性例子对模型进行微调。通过将对抗性图像纳入训练数据中，我们旨在防止偏见通过整个管道传播。我们通过定性和定量评估验证了我们的方法，证明了与现有方法相比，我们的方法具有更好的偏见缓解和准确性。定性结果表明，在训练后，模型的决策取决于敏感特征的程度，我们的模型更好地揭示了敏感特征和分类变量之间的关系。
URL
https://arxiv.org/abs/2404.11819
