Abstract
In this paper, we investigate the relationship between diversity metrics, accuracy, and resiliency to natural image corruptions of Deep Learning (DL) image classifier ensembles. We investigate the potential of an attribution-based diversity metric to improve the known accuracy-diversity trade-off of the typical prediction-based diversity. Our motivation is based on analytical studies of design diversity that have shown that a reduction of common failure modes is possible if diversity of design choices is achieved. Using ResNet50 as a comparison baseline, we evaluate the resiliency of multiple individual DL model architectures against dataset distribution shifts corresponding to natural image corruptions. We compare ensembles created with diverse model architectures trained either independently or through a Neural Architecture Search technique and evaluate the correlation of prediction-based and attribution-based diversity to the final ensemble accuracy. We evaluate a set of diversity enforcement heuristics based on negative correlation learning to assess the final ensemble resilience to natural image corruptions and inspect the resulting prediction, activation, and attribution diversity. Our key observations are: 1) model architecture is more important for resiliency than model size or model accuracy, 2) attribution-based diversity is less negatively correlated to the ensemble accuracy than prediction-based diversity, 3) a balanced loss function of individual and ensemble accuracy creates more resilient ensembles for image natural corruptions, 4) architecture diversity produces more diversity in all explored diversity metrics: predictions, attributions, and activations.
Abstract (translated)
在本文中,我们研究了深度学习(DL)图像分类器集体对自然图像 corruptions的 resilient 和多样性之间的关系。我们研究基于贡献( attribution)的多样性度量来改善典型的预测多样性中的准确性和多样性之间的权衡。我们的研究动力是基于设计多样性的分析研究,该研究已经表明,如果设计选择的多样性得到实现,可以减少常见的故障模式。使用ResNet50作为比较基准,我们评估了多个个体深度学习模型架构对数据集分布 shift 对应的自然图像 corruptions 的 resilient 能力。我们比较了由多种模型架构训练或通过神经网络架构搜索技术训练的集体,并评估预测和贡献多样性与最终集体准确性之间的相关性。我们评估了基于负相关学习的多样性执行技巧,以评估最终集体对自然图像 corruptions 的 resilient 能力,并检查产生预测、激活和贡献多样性的结果。我们的关键观察是:1) 模型架构对 resilient 能力的重要性比模型大小或模型精度更高,2) 贡献多样性与集体准确性之间的负相关性比预测多样性低,3) 个体和集体准确性的平衡损失函数创造了更抗损坏的集体对图像自然 corruptions,4) 架构多样性在所有探索的多样性度量中产生了更多的多样性:预测、贡献和激活。
URL
https://arxiv.org/abs/2303.09283
