Abstract
Open-vocabulary video visual relationship detection aims to detect objects and their relationships in videos without being restricted by predefined object or relationship categories. Existing methods leverage the rich semantic knowledge of pre-trained vision-language models such as CLIP to identify novel categories. They typically adopt a cascaded pipeline to first detect objects and then classify relationships based on the detected objects, which may lead to error propagation and thus suboptimal performance. In this paper, we propose Mutual EnhancemenT of Objects and Relationships (METOR), a query-based unified framework to jointly model and mutually enhance object detection and relationship classification in open-vocabulary scenarios. Under this framework, we first design a CLIP-based contextual refinement encoding module that extracts visual contexts of objects and relationships to refine the encoding of text features and object queries, thus improving the generalization of encoding to novel categories. Then we propose an iterative enhancement module to alternatively enhance the representations of objects and relationships by fully exploiting their interdependence to improve recognition performance. Extensive experiments on two public datasets, VidVRD and VidOR, demonstrate that our framework achieves state-of-the-art performance.
Abstract (translated)
开放词汇视频视觉关系检测的目标是在不受预定义对象或关系类别限制的情况下，识别视频中的物体及其关系。现有方法利用如CLIP等预训练的视觉-语言模型丰富的语义知识来识别新型类别。然而，这些方法通常采用级联流水线，先检测出物体再基于这些物体分类它们之间的关系，这种做法可能导致错误传播并导致性能欠佳。 本文中我们提出了互增强对象和关系框架（Mutual EnhancemenT of Objects and Relationships, METOR），这是一个以查询为基础的统一框架，在开放词汇场景下同时建模和相互增强目标检测与关系分类。在该框架内，首先设计了一个基于CLIP的上下文细化编码模块，用于提取物体和关系的视觉背景，并以此来改进文本特征和对象查询的编码，从而提高对新类别泛化的编码能力。然后提出了一种迭代增强模块，通过充分利用其相互依赖性交替地增强对象与关系的表现形式，以提升识别性能。 在两个公开数据集VidVRD和VidOR上进行广泛实验后证明了该框架实现了当前最优的性能。
URL
https://arxiv.org/abs/2505.06663
