Abstract
Visual relation detection (VRD) aims to identify relationships (or interactions) between object pairs in an image. Although recent VRD models have achieved impressive performance, they are all restricted to pre-defined relation categories, while failing to consider the semantic ambiguity characteristic of visual relations. Unlike objects, the appearance of visual relations is always subtle and can be described by multiple predicate words from different perspectives, e.g., ``ride'' can be depicted as ``race'' and ``sit on'', from the sports and spatial position views, respectively. To this end, we propose to model visual relations as continuous embeddings, and design diffusion models to achieve generalized VRD in a conditional generative manner, termed Diff-VRD. We model the diffusion process in a latent space and generate all possible relations in the image as an embedding sequence. During the generation, the visual and text embeddings of subject-object pairs serve as conditional signals and are injected via cross-attention. After the generation, we design a subsequent matching stage to assign the relation words to subject-object pairs by considering their semantic similarities. Benefiting from the diffusion-based generative process, our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets. To properly evaluate this generalized VRD task, we introduce two evaluation metrics, i.e., text-to-image retrieval and SPICE PR Curve inspired by image captioning. Extensive experiments in both human-object interaction (HOI) detection and scene graph generation (SGG) benchmarks attest to the superiority and effectiveness of Diff-VRD.
Abstract (translated)
视觉关系检测（VRD）的目标是识别图像中对象对之间的关系（或互动）。尽管最近的VRD模型已经取得了令人印象深刻的表现，但它们都局限于预定义的关系类别，并未能考虑视觉关系所具有的语义模糊性特征。与物体不同，视觉关系的外观总是微妙的，可以从不同的视角用多个谓词单词来描述，例如，“骑”可以分别从体育和空间位置的角度描绘为“比赛”和“坐在上面”。为此，我们提出将视觉关系建模为连续嵌入，并设计扩散模型以在条件生成方式下实现泛化的VRD，命名为Diff-VRD。我们在隐式空间中对扩散过程进行建模，并生成图像中的所有可能的关系作为嵌入序列。在生成过程中，主体-客体对的视觉和文本嵌入充当条件信号并通过交叉注意力机制注入其中。生成之后，我们设计了一个后续匹配阶段以根据它们的语义相似性将关系词分配给主体-客体对。得益于基于扩散的生成过程，我们的Diff-VRD能够生成超出数据集预定义类别标签的视觉关系。为了适当评估这项泛化的VRD任务，我们引入了两个评价指标，即文本到图像检索和灵感来自图像字幕的SPICE PR曲线。在人类对象交互（HOI）检测和场景图生成（SGG）基准测试中的广泛实验证明了Diff-VRD的优越性和有效性。
URL
https://arxiv.org/abs/2504.12100
