Abstract
Dynamic Scene Graph Generation (DSGG) aims to create a scene graph for each video frame by detecting objects and predicting their relationships. Weakly Supervised DSGG (WS-DSGG) reduces annotation workload by using an unlocalized scene graph from a single frame per video for training. Existing WS-DSGG methods depend on an off-the-shelf external object detector to generate pseudo labels for subsequent DSGG training. However, detectors trained on static, object-centric images struggle in dynamic, relation-aware scenarios required for DSGG, leading to inaccurate localization and low-confidence proposals. To address the challenges posed by external object detectors in WS-DSGG, we propose a Temporal-enhanced Relation-aware Knowledge Transferring (TRKT) method, which leverages knowledge to enhance detection in relation-aware dynamic scenarios. TRKT is built on two key components:(1)Relation-aware knowledge mining: we first employ object and relation class decoders that generate category-specific attention maps to highlight both object regions and interactive areas. Then we propose an Inter-frame Attention Augmentation strategy that exploits optical flow for neighboring frames to enhance the attention maps, making them motion-aware and robust to motion blur. This step yields relation- and motion-aware knowledge mining for WS-DSGG. (2) we introduce a Dual-stream Fusion Module that integrates category-specific attention maps into external detections to refine object localization and boost confidence scores for object proposals. Extensive experiments demonstrate that TRKT achieves state-of-the-art performance on Action Genome dataset. Our code is avaliable at this https URL.
Abstract (translated)
动态场景图生成（DSGG）的目标是通过检测物体并预测它们之间的关系，为视频中的每一帧创建一个场景图。弱监督的动态场景图生成（WS-DSGG）方法通过使用来自每个视频单个帧的未定位的场景图来减少标注工作量。现有的WS-DSGG方法依赖于现成的外部对象检测器来生成用于后续DSGG训练的伪标签。然而，基于静态、以物体为中心的图像进行训练的对象检测器在动态关系感知场景下表现不佳，导致定位不准确和低置信度建议。 为了解决WS-DSGG中外部对象检测器带来的挑战，我们提出了一种时间增强的关系感知知识迁移（TRKT）方法，该方法利用知识来提高在关系感知动态场景中的检测性能。TRKT建立在两个关键组件上： 1. **关系感知的知识挖掘**：首先使用物体和关系类解码器生成特定于类别的注意图以突出显示对象区域及其互动区。然后我们提出了一种跨帧注意力增强策略，利用光流来提高相邻帧中的注意图，使其能够感知运动，并且在面对运动模糊时更为鲁棒。这一步产生了适用于WS-DSGG的关系和运动感知的知识挖掘。 2. **双流融合模块**：此模块将特定于类别的注意图集成到外部检测结果中，以细化物体定位并提高对象建议的置信度得分。 广泛的实验表明，TRKT在Action Genome数据集上达到了最先进的性能。我们的代码可以在这里访问（原文中的URL链接）。
URL
https://arxiv.org/abs/2508.04943
