Abstract
Scene graph generation (SGG) and human-object interaction (HOI) detection are two important visual tasks aiming at localising and recognising relationships between objects, and interactions between humans and objects, respectively. Prevailing works treat these tasks as distinct tasks, leading to the development of task-specific models tailored to individual datasets. However, we posit that the presence of visual relationships can furnish crucial contextual and intricate relational cues that significantly augment the inference of human-object interactions. This motivates us to think if there is a natural intrinsic relationship between the two tasks, where scene graphs can serve as a source for inferring human-object interactions. In light of this, we introduce SG2HOI+, a unified one-step model based on the Transformer architecture. Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection. Concretely, we initiate a relation Transformer tasked with generating relation triples from a suite of visual features. Subsequently, we employ another transformer-based decoder to predict human-object interactions based on the generated relation triples. A comprehensive series of experiments conducted across established benchmark datasets including Visual Genome, V-COCO, and HICO-DET demonstrates the compelling performance of our SG2HOI+ model in comparison to prevalent one-stage SGG models. Remarkably, our approach achieves competitive performance when compared to state-of-the-art HOI methods. Additionally, we observe that our SG2HOI+ jointly trained on both SGG and HOI tasks in an end-to-end manner yields substantial improvements for both tasks compared to individualized training paradigms.
Abstract (translated)
场景图生成（SGG）和人类-物体交互（HOI）检测是旨在定位和识别物体之间关系以及人类与物体之间关系的两个重要视觉任务。先前的研究将这些任务视为独立的任务，导致针对单个数据集开发了特定任务的模型。然而，我们提出，视觉关系的存在可以为人类-物体交互推理提供关键的上下文和复杂的关系线索，从而显著增强交互推断。这激励我们将目光放在这两个任务之间是否存在自然内在关系上，其中场景图可以作为推断人类-物体交互的来源。因此，我们引入了SG2HOI+，一种基于Transformer架构的统一一步模型。我们的方法采用两个交互式分层Transformer来无缝统一SGG和HOI检测任务。具体来说，我们使用关系Transformer生成一系列视觉特征中的一对关系三元组。然后，我们使用另一个基于Transformer的解码器根据生成的关系三元组预测人类-物体交互。在包括Visual Genome、V-COCO和HICO-DET等现有基准数据集的全面系列实验中，展示了我们SG2HOI+模型的引人入胜的性能与先前的单阶段SGG模型的性能相比。值得注意的是，与最先进的HOI方法相比，我们的方法在性能上具有竞争优势。此外，我们观察到，在端到端的方式同时训练SGG和HOI任务时，我们的SG2HOI+模型对于两个任务都取得了显著的提高，相对于个性化的训练范式。
URL
https://arxiv.org/abs/2311.01755
