Abstract
Real depth super-resolution (DSR), unlike synthetic settings, is a challenging task due to the structural distortion and the edge noise caused by the natural degradation in real-world low-resolution (LR) depth maps. These defeats result in significant structure inconsistency between the depth map and the RGB guidance, which potentially confuses the RGB-structure guidance and thereby degrades the DSR quality. In this paper, we propose a novel structure flow-guided DSR framework, where a cross-modality flow map is learned to guide the RGB-structure information transferring for precise depth upsampling. Specifically, our framework consists of a cross-modality flow-guided upsampling network (CFUNet) and a flow-enhanced pyramid edge attention network (PEANet). CFUNet contains a trilateral self-attention module combining both the geometric and semantic correlations for reliable cross-modality flow learning. Then, the learned flow maps are combined with the grid-sampling mechanism for coarse high-resolution (HR) depth prediction. PEANet targets at integrating the learned flow map as the edge attention into a pyramid network to hierarchically learn the edge-focused guidance feature for depth edge refinement. Extensive experiments on real and synthetic DSR datasets verify that our approach achieves excellent performance compared to state-of-the-art methods.
Abstract (translated)
与合成设置不同,真实的深度超分辨率(DSR)是一个挑战性的任务,因为真实的低分辨率(LR)深度地图由于自然退化的原因发生了结构扭曲和边缘噪声。这些失败会导致深度地图和RGB guidance之间的结构不一致,这可能会混淆RGB结构 guidance,从而降低DSR质量。在本文中,我们提出了一种结构引导的DSR框架,其中学习交叉modality flow map来指导RGB结构信息传输,以进行精确的深度增广。具体来说,我们的框架由一个交叉modality flow引导增广网络(Cfunet)和一个流增强金字塔边缘注意力网络(PEANet)组成。Cfunet包含一个三角形自注意力模块,结合几何和语义corr关系,以可靠地学习交叉modality flow learning。然后,学习的流程图与网格采样机制一起用于粗级高分辨率(HR)深度预测。PEANet的目标是将学习的流程图作为金字塔网络的边缘注意力集成起来,以Hierarchically learn edge-focused guidance feature for depth edge refinement。对真实和合成的DSR数据集进行的广泛实验证实,我们的方法相对于先进的方法取得了出色的表现。
URL
https://arxiv.org/abs/2301.13416
