Abstract
Visual relation detection (VRD) is the task of identifying the relationships between objects in a scene. VRD models trained solely on relation detection data struggle to generalize beyond the relations on which they are trained. While prompt tuning has been used to adapt vision-language models (VLMs) for VRD, it uses handcrafted prompts and struggles with novel or complex relations. We argue that instruction tuning offers a more effective solution by fine-tuning VLMs on diverse instructional data. We thus introduce ART, an Adaptive Relation Tuning framework that adapts VLMs for VRD through instruction tuning and strategic instance selection. By converting VRD datasets into an instruction tuning format and employing an adaptive sampling algorithm, ART directs the VLM to focus on informative relations while maintaining generalizability. Specifically, we focus on the relation classification, where subject-object boxes are given and the model predicts the predicate between them. We tune on a held-in set and evaluate across multiple held-out datasets of varying complexity. Our approach strongly improves over its baselines and can infer unseen relation concepts, a capability absent in mainstream VRD methods. We demonstrate ART's practical value by using the predicted relations for segmenting complex scenes.
Abstract (translated)
视觉关系检测（VRD）的任务是识别场景中物体之间的关系。仅基于关系检测数据训练的VRD模型难以泛化到它们未见过的关系上。虽然提示微调已经被用来适应视觉-语言模型（VLMs）以进行VRD，但这种方法使用的是手工设计的提示，并且在处理新颖或复杂的关系时效果不佳。我们认为，指令微调提供了一种更为有效的解决方案，通过在多样化的指令数据上对VLM进行精细调整来实现这一目标。 因此，我们引入了ART（自适应关系微调框架），该框架利用指令微调和策略性实例选择的方式将VLM适配到VRD中。通过将VRD数据集转换为指令微调格式，并使用自适应采样算法，ART指导模型关注具有信息性的关系同时保持泛化能力。 具体而言，我们专注于关系分类任务，在这个任务中给出了主体-客体框，模型需要预测它们之间的谓词。我们在预留的训练集上进行微调，并在多个不同复杂度级别的未见过的数据集上进行了评估。我们的方法显著优于基线方法，并且能够推理出未曾见过的关系概念，这是主流VRD方法所不具备的能力。 我们通过使用预测关系来分割复杂的场景展示了ART的实际应用价值。
URL
https://arxiv.org/abs/2507.23543
