Abstract
Applying salient object detection (SOD) to RGB-D videos is an emerging task called RGB-D VSOD and has recently gained increasing interest, due to considerable performance gains of incorporating motion and depth and that RGB-D videos can be easily captured now in daily life. Existing RGB-D VSOD models have different attempts to derive motion cues, in which extracting motion information explicitly from optical flow appears to be a more effective and promising alternative. Despite this, there remains a key issue that how to effectively utilize optical flow and depth to assist the RGB modality in SOD. Previous methods always treat optical flow and depth equally with respect to model designs, without explicitly considering their unequal contributions in individual scenarios, limiting the potential of motion and depth. To address this issue and unleash the power of motion and depth, we propose a novel selective cross-modal fusion framework (SMFNet) for RGB-D VSOD, incorporating a pixel-level selective fusion strategy (PSF) that achieves optimal fusion of optical flow and depth based on their actual contributions. Besides, we propose a multi-dimensional selective attention module (MSAM) to integrate the fused features derived from PSF with the remaining RGB modality at multiple dimensions, effectively enhancing feature representation to generate refined features. We conduct comprehensive evaluation of SMFNet against 19 state-of-the-art models on both RDVS and DVisal datasets, making the evaluation the most comprehensive RGB-D VSOD benchmark up to date, and it also demonstrates the superiority of SMFNet over other models. Meanwhile, evaluation on five video benchmark datasets incorporating synthetic depth validates the efficacy of SMFNet as well. Our code and benchmark results are made publicly available at this https URL.
Abstract (translated)
将显著目标检测（SOD）应用于RGB-D视频是一种新兴的任务，被称为RGB-D VSOD（Video Salient Object Detection），由于结合运动和深度信息可以显著提高性能，并且现在日常生活中很容易捕捉到RGB-D视频，因此最近引起了越来越多的关注。现有的RGB-D VSOD模型在推导运动线索方面有不同的尝试，其中从光流中显式提取运动信息似乎是一种更有效且有前景的替代方法。尽管如此，仍然存在一个关键问题是如何有效地利用光流和深度来辅助RGB模态进行SOD。先前的方法总是将光流和深度在模型设计上等同对待，没有明确考虑它们在不同场景中的贡献差异，这限制了运动和深度的潜力。 为了解决这一问题并释放运动和深度的力量，我们提出了一种新颖的选择性跨模式融合框架（SMFNet），用于RGB-D VSOD。该框架结合了一个像素级别的选择性融合策略（PSF），基于光流和深度的实际贡献实现了最优融合。此外，我们还提出了一个多维选择性注意模块（MSAM），以在多个维度上将通过PSF获得的融合特征与剩余的RGB模态集成起来，从而有效增强特征表示生成精炼的特征。 我们在RDVS和DVisal数据集上对SMFNet进行了全面评估，使其成为迄今为止最全面的RGB-D VSOD基准测试，并且它也展示了SMFNet相对于其他模型的优势。同时，在包含合成深度的五个视频基准测试数据集上的评估验证了SMFNet的有效性。 我们的代码和基准结果公开提供在以下网址：[此链接]（请将方括号中的内容替换为实际提供的URL）。
URL
https://arxiv.org/abs/2507.21857
