Abstract
We present a novel self-supervised approach for representation learning, particularly for the task of Visual Relationship Detection (VRD). Motivated by the effectiveness of Masked Image Modeling (MIM), we propose Masked Bounding Box Reconstruction (MBBR), a variation of MIM where a percentage of the entities/objects within a scene are masked and subsequently reconstructed based on the unmasked objects. The core idea is that, through object-level masked modeling, the network learns context-aware representations that capture the interaction of objects within a scene and thus are highly predictive of visual object relationships. We extensively evaluate learned representations, both qualitatively and quantitatively, in a few-shot setting and demonstrate the efficacy of MBBR for learning robust visual representations, particularly tailored for VRD. The proposed method is able to surpass state-of-the-art VRD methods on the Predicate Detection (PredDet) evaluation setting, using only a few annotated samples. We make our code available at this https URL.
Abstract (translated)
我们提出了一个新颖的自监督表示学习方法，特别是针对视觉关系检测（VRD）任务。受到掩码图像建模（MIM）的有效性的启发，我们提出了掩码边界框重构（MBBR），这是一种MIM的变体，其中场景中的实体/对象的一部分被遮罩，然后根据未遮罩的对象进行重构。核心思想是，通过物体级别的遮罩建模，网络学习了一个捕捉场景中物体之间交互的上下文感知表示，因此具有高度预测视觉对象关系的预测能力。我们在几个样本设置中对其学习到的表示进行了广泛的评估，并且证明了MBBR对于学习适用于VRD的稳健表示非常有效。该方法在仅用几篇注释样本的情况下，能够超越最先进的VRD方法在命题检测（PredDet）评估设置。我们将代码公开在以下链接处：
URL
https://arxiv.org/abs/2311.04834
