Abstract
Visual relationship understanding has been studied separately in human-object interaction(HOI) detection, scene graph generation(SGG), and referring relationships(RR) tasks. Given the complexity and interconnectedness of these tasks, it is crucial to have a flexible framework that can effectively address these tasks in a cohesive manner. In this work, we propose FleVRS, a single model that seamlessly integrates the above three aspects in standard and promptable visual relationship segmentation, and further possesses the capability for open-vocabulary segmentation to adapt to novel scenarios. FleVRS leverages the synergy between text and image modalities, to ground various types of relationships from images and use textual features from vision-language models to visual conceptual understanding. Empirical validation across various datasets demonstrates that our framework outperforms existing models in standard, promptable, and open-vocabulary tasks, e.g., +1.9 $mAP$ on HICO-DET, +11.4 $Acc$ on VRD, +4.7 $mAP$ on unseen HICO-DET. Our FleVRS represents a significant step towards a more intuitive, comprehensive, and scalable understanding of visual relationships.
Abstract (translated)
视觉关系理解已经在人机交互（HOI）检测、场景图生成（SGG）和引用关系（RR）任务中进行了单独研究。考虑到这些任务的复杂性和相互关联性，有必要提供一个灵活的框架，能够以一种集成的方式有效地解决这些任务。在本文中，我们提出了FleVRS，一种将上述三个方面无缝集成在标准和提示性视觉关系分割中的单一模型，并进一步具有适应新场景的开放词汇分割能力。FleVRS利用文本和图像模态之间的协同作用，将各种类型的关系从图像中 grounded，并使用视觉语言模型中的文本特征进行视觉概念理解。通过不同数据集的实证验证，我们的框架在标准、提示性和开放词汇任务中优于现有模型，例如+1.9 $mAP$在HICO-DET，+11.4 $Acc$在VRD，+4.7 $mAP$在未见过的HICO-DET上。FleVRS代表了一个明显朝着更直观、全面和可扩展的视觉关系理解方向迈出的重要一步。
URL
https://arxiv.org/abs/2408.08305
