Abstract
Recent unsupervised video object segmentation (UVOS) methods predominantly adopt the motion-appearance paradigm. Mainstream motion-appearance approaches use either the two-encoder structure to separately encode motion and appearance features, or the single-encoder structure for joint encoding. However, these methods fail to properly balance the motion-appearance relationship. Consequently, even with complex fusion modules for motion-appearance integration, the extracted suboptimal features degrade the models' overall performance. Moreover, the quality of optical flow varies across scenarios, making it insufficient to rely solely on optical flow to achieve high-quality segmentation results. To address these challenges, we propose the Intrinsic Saliency guided Trunk-Collateral Net}work (ISTC-Net), which better balances the motion-appearance relationship and incorporates model's intrinsic saliency information to enhance segmentation performance. Specifically, considering that optical flow maps are derived from RGB images, they share both commonalities and differences. We propose a novel Trunk-Collateral structure. The shared trunk backbone captures the motion-appearance commonality, while the collateral branch learns the uniqueness of motion features. Furthermore, an Intrinsic Saliency guided Refinement Module (ISRM) is devised to efficiently leverage the model's intrinsic saliency information to refine high-level features, and provide pixel-level guidance for motion-appearance fusion, thereby enhancing performance without additional input. Experimental results show that ISTC-Net achieved state-of-the-art performance on three UVOS datasets (89.2% J&F on DAVIS-16, 76% J on YouTube-Objects, 86.4% J on FBMS) and four standard video salient object detection (VSOD) benchmarks with the notable increase, demonstrating its effectiveness and superiority over previous methods.
Abstract (translated)
最近的无监督视频对象分割（UVOS）方法主要采用运动外观范式。主流的运动外观方法要么使用双编码器结构来分别编码运动和外观特征，要么使用单个编码器进行联合编码。然而，这些方法未能很好地平衡运动与外观之间的关系。因此，即使配备了复杂的融合模块用于整合运动与外观，提取出的次优特征仍然会降低模型的整体性能。此外，光流的质量在不同的场景中变化很大，这使得仅依赖于光流来实现高质量分割结果变得不够充分。 为了解决这些挑战，我们提出了基于内在显著性引导的主干分支网络（ISTC-Net）。该方法能够更好地平衡运动与外观之间的关系，并通过利用模型的内在显著性信息来提升分割性能。具体而言，考虑到光学流图是从RGB图像中推导出来的，它们既有共同之处也有差异点。为此，我们设计了一种新颖的主干分支结构：共享的主干背骨捕捉了运动和外观的共通性，而分支部分则学习到运动特征的独特性。 此外，还开发了一个基于内在显著性的精炼模块（ISRM），旨在有效地利用模型的内在显著信息来细化高层次特征，并为运动与外观融合提供像素级别的指导，从而在不增加额外输入的情况下提高性能。 实验结果显示，ISTC-Net 在三个 UVOS 数据集（DAVIS-16 的 J&F 评分达到 89.2%，YouTube-Objects 的 J 评分为 76%，FBMS 的 J 评分为 86.4%）以及四个标准视频显著对象检测（VSOD）基准测试中取得了最先进的性能，并且与之前的模型相比有明显提升，这证明了其有效性及优越性。
URL
https://arxiv.org/abs/2504.05904
