Abstract
Open-vocabulary Scene Graph Generation (OV-SGG) overcomes the limitations of the closed-set assumption by aligning visual relationship representations with open-vocabulary textual representations. This enables the identification of novel visual relationships, making it applicable to real-world scenarios with diverse relationships. However, existing OV-SGG methods are constrained by fixed text representations, limiting diversity and accuracy in image-text alignment. To address these challenges, we propose the Relation-Aware Hierarchical Prompting (RAHP) framework, which enhances text representation by integrating subject-object and region-specific relation information. Our approach utilizes entity clustering to address the complexity of relation triplet categories, enabling the effective integration of subject-object information. Additionally, we utilize a large language model (LLM) to generate detailed region-aware prompts, capturing fine-grained visual interactions and improving alignment between visual and textual modalities. RAHP also introduces a dynamic selection mechanism within Vision-Language Models (VLMs), which adaptively selects relevant text prompts based on the visual content, reducing noise from irrelevant prompts. Extensive experiments on the Visual Genome and Open Images v6 datasets demonstrate that our framework consistently achieves state-of-the-art performance, demonstrating its effectiveness in addressing the challenges of open-vocabulary scene graph generation.
Abstract (translated)
开放词汇场景图生成（OV-SGG）通过将视觉关系表示与开放式词汇文本表示对齐，克服了闭集假设的限制。这使得识别新颖的视觉关系成为可能，并使其适用于具有多样化关系的真实世界场景。然而，现有的OV-SGG方法受到固定文本表示方式的约束，这对图像和文本之间的多样性和准确性匹配造成了限制。 为了应对这些挑战，我们提出了基于关系感知分层提示（RAHP）框架，该框架通过整合主体-客体及区域特定的关系信息来增强文本表示。我们的方法利用实体聚类来处理关系三元组类别复杂性的问题，并有效地集成主体-客体信息。此外，我们还使用大型语言模型（LLM）生成详细且区域感知的提示词，捕捉细微的视觉互动并提高视觉与文本模态之间的对齐度。 RAHP框架还引入了在视觉-语言模型（VLMs）中的动态选择机制，该机制根据视觉内容自适应地选择相关文本提示，从而减少了无关提示造成的噪声干扰。我们在Visual Genome和Open Images v6数据集上的广泛实验表明，我们的框架能够持续达到最先进的性能水平，证明了它在解决开放词汇场景图生成挑战方面的有效性。 综上所述，通过提出基于关系感知分层提示（RAHP）的框架，我们不仅增强了文本表示的能力，还提高了视觉-语言模型在处理复杂、多样化场景时的表现，从而为开放词汇场景图生成提供了一种有效的解决方案。
URL
https://arxiv.org/abs/2412.19021
