Abstract
We propose an adaptation to the training of Vision Transformers (ViTs) that allows for an explicit modeling of objects during the attention computation. This is achieved by adding a new branch to selected attention layers that computes an auxiliary loss which we call the object-focused attention (OFA) loss. We restrict the attention to image patches that belong to the same object class, which allows ViTs to gain a better understanding of configural (or holistic) object shapes by focusing on intra-object patches instead of other patches such as those in the background. Our proposed inductive bias fits easily into the attention framework of transformers since it only adds an auxiliary loss over selected attention layers. Furthermore, our approach has no additional overhead during inference. We also experiment with multiscale masking to further improve the performance of our OFA model and give a path forward for self-supervised learning with our method. Our experimental results demonstrate that ViTs with OFA achieve better classification results than their base models, exhibit a stronger generalization ability to out-of-distribution (OOD) and adversarially corrupted images, and learn representations based on object shapes rather than spurious correlations via general textures. For our OOD setting, we generate a novel dataset using the COCO dataset and Stable Diffusion inpainting which we plan to share with the community.
Abstract (translated)
我们提出了一种对视觉变换器（ViT）训练的适应性调整，该调整允许在注意力计算过程中显式地对物体进行建模。通过向选定的关注层添加一个新的分支来实现这一点，这个新分支计算一个辅助损失，即所谓的对象聚焦关注（OFA）损失。我们将注意力限制在属于同一物体类别的图像补丁上，这使ViT能够更好地理解配置图（或整体）的物体形状，方法是将重点放在物体内部的补丁上，而不是背景中的其他补丁。我们提出的归纳偏差很容易融入到变换器的关注框架中，因为它仅向选定的关注层添加了一个辅助损失，并且在推理过程中没有额外的开销。此外，我们还通过多尺度掩码实验来进一步提高我们的OFA模型的表现，并为使用我们方法进行自监督学习提供了一条路径。实验证明了，带有OFA的ViT比其基础模型有更好的分类效果，更强地泛化到分布外（OOD）和对抗性损坏的图像上，并且基于物体形状而非偶然关联的一般纹理来学习表示。对于我们的OOD设置，我们使用COCO数据集和Stable Diffusion inpainting生成了一个新的数据集，计划与社区分享。
URL
https://arxiv.org/abs/2504.08166
