Abstract
Dynamic Scene Graph Generation (DSGG) focuses on identifying visual relationships within the spatial-temporal domain of videos. Conventional approaches often employ multi-stage pipelines, which typically consist of object detection, temporal association, and multi-relation classification. However, these methods exhibit inherent limitations due to the separation of multiple stages, and independent optimization of these sub-problems may yield sub-optimal solutions. To remedy these limitations, we propose a one-stage end-to-end framework, termed OED, which streamlines the DSGG pipeline. This framework reformulates the task as a set prediction problem and leverages pair-wise features to represent each subject-object pair within the scene graph. Moreover, another challenge of DSGG is capturing temporal dependencies, we introduce a Progressively Refined Module (PRM) for aggregating temporal context without the constraints of additional trackers or handcrafted trajectories, enabling end-to-end optimization of the network. Extensive experiments conducted on the Action Genome benchmark demonstrate the effectiveness of our design. The code and models are available at \url{this https URL}.
Abstract (translated)
动态场景图生成（DSGG）关注视频的空间-时间域内的视觉关系。传统的解决方案通常采用多阶段流程，通常包括目标检测、时间关联和多关系分类。然而，由于多个阶段的分离，这些方法存在固有局限性，而独立优化这些子问题可能会产生次优解决方案。为了弥补这些局限性，我们提出了一个端到端的框架，称为OED，该框架简化了DSGG管道。该框架将任务重新建模为预测问题，并利用成对特征表示场景图中的每个主题-对象对。此外，DSGG的一个挑战是捕捉时间依赖关系，我们引入了一个逐渐精炼的模块（PRM），用于汇总没有额外跟踪器或手工制作的轨迹的元数据，从而实现网络端到端的优化。在Action Genome基准上进行的大量实验证明了我们设计的有效性。代码和模型可在此处访问：\url{这个链接}。
URL
https://arxiv.org/abs/2405.16925
