Abstract
Visual relationship detection aims to identify objects and their relationships in images. Prior methods approach this task by adding separate relationship modules or decoders to existing object detection architectures. This separation increases complexity and hinders end-to-end training, which limits performance. We propose a simple and highly efficient decoder-free architecture for open-vocabulary visual relationship detection. Our model consists of a Transformer-based image encoder that represents objects as tokens and models their relationships implicitly. To extract relationship information, we introduce an attention mechanism that selects object pairs likely to form a relationship. We provide a single-stage recipe to train this model on a mixture of object and relationship detection data. Our approach achieves state-of-the-art relationship detection performance on Visual Genome and on the large-vocabulary GQA benchmark at real-time inference speeds. We provide analyses of zero-shot performance, ablations, and real-world qualitative examples.
Abstract (translated)
视觉关系检测旨在在图像中识别物体及其关系。之前的方法通过向现有的物体检测架构中添加单独的关系模块或解码器来解决这个问题。这种分离增加了复杂度，并阻碍了端到端训练，这限制了性能。我们提出了一种简单的且高效的无解码器架构，用于开放词汇的视觉关系检测。我们的模型包括一个基于Transformer的图像编码器，它将物体表示为标记，并隐含地建模它们之间的关系。为了提取关系信息，我们引入了一个注意力机制，选择可能形成关系的物体对。我们提供了一种在混合物体和关系检测数据上训练此模型的单阶段 recipe。我们的方法在实时推理速度下实现了视觉基因组和大型词汇GQA基准中的最先进关系检测性能。我们还提供了关于零散性能、消融和真实世界质量实例的分析。
URL
https://arxiv.org/abs/2403.14270
