Abstract
We present Rewis3d, a framework that leverages recent advances in feed-forward 3D reconstruction to significantly improve weakly supervised semantic segmentation on 2D images. Obtaining dense, pixel-level annotations remains a costly bottleneck for training segmentation models. Alleviating this issue, sparse annotations offer an efficient weakly-supervised alternative. However, they still incur a performance gap. To address this, we introduce a novel approach that leverages 3D scene reconstruction as an auxiliary supervisory signal. Our key insight is that 3D geometric structure recovered from 2D videos provides strong cues that can propagate sparse annotations across entire scenes. Specifically, a dual student-teacher architecture enforces semantic consistency between 2D images and reconstructed 3D point clouds, using state-of-the-art feed-forward reconstruction to generate reliable geometric supervision. Extensive experiments demonstrate that Rewis3d achieves state-of-the-art performance in sparse supervision, outperforming existing approaches by 2-7% without requiring additional labels or inference overhead.
Abstract (translated)
我们介绍了一个名为Rewis3d的框架，该框架利用近期在前馈式三维重建方面的进展，显著提升了二维图像上弱监督语义分割的效果。获得密集、像素级别的注释仍然是训练分割模型的一个昂贵瓶颈问题。为了解决这个问题，稀疏标注提供了一种有效的弱监督替代方案。然而，这种做法仍然会导致性能差距。为此，我们引入了一个新颖的方法，该方法利用三维场景重建作为辅助监督信号。 我们的关键见解是，从二维视频中恢复出的三维几何结构可以提供强大的线索，从而将稀疏注释传播到整个场景中。具体来说，一个双学生的教师架构强制执行二维图像和重构的三维点云之间的语义一致性，并使用最先进的前馈式重建来生成可靠的几何监督。 广泛的实验表明，Rewis3d在稀疏监督下达到了最先进的性能，相比现有方法提高了2-7%，并且不需要额外的标签或推理开销。
URL
https://arxiv.org/abs/2603.06374
