Abstract
Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details. However, the misleading color and texture cues indicating depth discontinuities in RGB images often lead to artifacts and blurred depth boundaries in the generated depth map. We propose a solution that introduces global contextual semantic priors, generated from pretrained vision transformer token embeddings. Our approach to distilling semantic knowledge from pretrained token embeddings is motivated by their demonstrated effectiveness in related monocular depth estimation tasks. We introduce a Guided Token Attention (GTA) module, which iteratively aligns encoded RGB spatial features with depth encodings, using cross-attention for selectively injecting global semantic context extracted from different layers of a pretrained vision transformer. Additionally, we present an architecture called Neural Attention for Implicit Multi-token Alignment (NAIMA), which integrates DINOv2 with GTA blocks for a semantics-aware GDSR. Our proposed architecture, with its ability to distill semantic knowledge, achieves significant improvements over existing methods across multiple scaling factors and datasets.
Abstract (translated)
引导式深度超分辨率（GDSR）是一种基于多模态的深度图超分辨率方法，它依赖低分辨率深度图与高分辨率RGB图像来恢复更精细的结构细节。然而，RGB图像中误导性的颜色与纹理线索常导致深度不连续处的伪影与模糊边界。我们提出一种引入全局上下文语义先验的解决方案，该先验源自预训练视觉Transformer的标记嵌入。我们蒸馏预训练标记嵌入中语义知识的方法，受其在相关单目深度估计任务中已验证的有效性所启发。我们设计了引导式标记注意力（GTA）模块，通过交叉注意力机制迭代对齐RGB空间编码特征与深度编码，从而选择性注入从预训练视觉Transformer不同层提取的全局语义上下文。此外，我们提出了神经注意力隐式多标记对齐（NAIMA）架构，该架构将DINOv2与GTA模块集成，实现语义感知的GDSR。我们提出的架构凭借其语义知识蒸馏能力，在多种缩放因子与数据集上相较于现有方法取得了显著提升。
URL
https://arxiv.org/abs/2604.04407
