Abstract
Depth map enhancement using paired high-resolution RGB images offers a cost-effective solution for improving low-resolution depth data from lightweight ToF sensors. Nevertheless, naively adopting a depth estimation pipeline to fuse the two modalities requires groundtruth depth maps for supervision. To address this, we propose a self-supervised learning framework, SelfToF, which generates detailed and scale-aware depth maps. Starting from an image-based self-supervised depth estimation pipeline, we add low-resolution depth as inputs, design a new depth consistency loss, propose a scale-recovery module, and finally obtain a large performance boost. Furthermore, since the ToF signal sparsity varies in real-world applications, we upgrade SelfToF to SelfToF* with submanifold convolution and guided feature fusion. Consequently, SelfToF* maintain robust performance across varying sparsity levels in ToF data. Overall, our proposed method is both efficient and effective, as verified by extensive experiments on the NYU and ScanNet datasets. The code will be made public.
Abstract (translated)
使用配对的高分辨率RGB图像来增强深度图，为改善来自轻量级ToF（飞行时间）传感器的低分辨率深度数据提供了一种成本效益高的解决方案。然而，简单地采用传统的深度估计管道融合这两种模态需要地面实况深度图来进行监督。为了应对这一挑战，我们提出了一种自监督学习框架SelfToF，该框架能够生成详细且尺度感知的深度图。从基于图像的自监督深度估计管道开始，我们将低分辨率深度图作为输入加入进来，并设计了新的深度一致性损失函数，提出了尺度恢复模块，最终实现了性能的重大提升。 由于实际应用中ToF信号稀疏性会有所不同，我们进一步升级SelfToF框架为SelfToF*版本，通过添加子流形卷积和引导特征融合来适应不同情况。这样，SelfToF*在处理不同稀疏度级别的ToF数据时能够保持稳健的性能表现。 总的来说，我们的方法既高效又有效，这一点已经在NYU（纽约大学）和ScanNet数据集上的广泛实验中得到了验证。代码将在后续公开发布。
URL
https://arxiv.org/abs/2506.13444
