Abstract
Cross-spectral image guided denoising has shown its great potential in recovering clean images with rich details, such as using the near-infrared image to guide the denoising process of the visible one. To obtain such image pairs, a feasible and economical way is to employ a stereo system, which is widely used on mobile devices. Current works attempt to generate an aligned guidance image to handle the disparity between two images. However, due to occlusion, spectral differences and noise degradation, the aligned guidance image generally exists ghosting and artifacts, leading to an unsatisfactory denoised result. To address this issue, we propose a one-stage transformer-based architecture, named SGDFormer, for cross-spectral Stereo image Guided Denoising. The architecture integrates the correspondence modeling and feature fusion of stereo images into a unified network. Our transformer block contains a noise-robust cross-attention (NRCA) module and a spatially variant feature fusion (SVFF) module. The NRCA module captures the long-range correspondence of two images in a coarse-to-fine manner to alleviate the interference of noise. The SVFF module further enhances salient structures and suppresses harmful artifacts through dynamically selecting useful information. Thanks to the above design, our SGDFormer can restore artifact-free images with fine structures, and achieves state-of-the-art performance on various datasets. Additionally, our SGDFormer can be extended to handle other unaligned cross-model guided restoration tasks such as guided depth super-resolution.
Abstract (translated)
跨谱图像指导去噪在恢复清晰图像和丰富细节方面显示出巨大的潜力，例如利用近红外图像指导可见图像的降噪过程。为了获得这样的图像对，采用一个经济且可行的方式是使用双目立体系统，这已经在移动设备上得到了广泛应用。当前的工作试图生成一个对齐的指导图像来处理两张图像之间的差异。然而，由于遮挡、光谱差异和噪声衰减，对齐的指导图像通常存在伪影和伪色，导致去噪效果不满意。为了解决这个问题，我们提出了一个基于Transformer的一阶架构，称为SGDFormer，用于跨谱立体图像指导去噪。该架构将立体图像的对应关系建模和特征融合统一到一个网络中。我们的Transformer模块包含一个噪声鲁棒跨注意力（NRCA）模块和一个空间可变特征融合（SVFF）模块。NRCA模块以粗到细的方式捕捉两张图像之间的长距离对应关系，减轻噪声干扰。SVFF模块通过动态选择有用的信息来增强显著结构并抑制有害伪影。得益于上述设计，我们的SGDFormer可以恢复无伪影的图像，具有丰富的细节，并在各种数据集上实现最先进的性能。此外，我们的SGDFormer还可以扩展到处理其他未对齐的跨模型指导修复任务，如指导深度超分辨率。
URL
https://arxiv.org/abs/2404.00349
