Abstract
In this paper, we consider two challenging issues in reference-based super-resolution (RefSR) for smartphone, (i) how to choose a proper reference image, and (ii) how to learn RefSR in a self-supervised manner. Particularly, we propose a novel self-supervised learning approach for real-world RefSR from observations at dual and multiple camera zooms. Firstly, considering the popularity of multiple cameras in modern smartphones, the more zoomed (telephoto) image can be naturally leveraged as the reference to guide the super-resolution (SR) of the lesser zoomed (ultra-wide) image, which gives us a chance to learn a deep network that performs SR from the dual zoomed observations (DZSR). Secondly, for self-supervised learning of DZSR, we take the telephoto image instead of an additional high-resolution image as the supervision information, and select a center patch from it as the reference to super-resolve the corresponding ultra-wide image patch. To mitigate the effect of the misalignment between ultra-wide low-resolution (LR) patch and telephoto ground-truth (GT) image during training, we first adopt patch-based optical flow alignment and then design an auxiliary-LR to guide the deforming of the warped LR features. To generate visually pleasing results, we present local overlapped sliced Wasserstein loss to better represent the perceptual difference between GT and output in the feature space. During testing, DZSR can be directly deployed to super-solve the whole ultra-wide image with the reference of the telephoto image. In addition, we further take multiple zoomed observations to explore self-supervised RefSR, and present a progressive fusion scheme for the effective utilization of reference images. Experiments show that our methods achieve better quantitative and qualitative performance against state-of-the-arts. Codes are available at this https URL.
Abstract (translated)
在本文中，我们考虑了在基于参考图像的超分辨率（RefSR）中 two 个具有挑战性的问题：（i）如何选择一个适当的参考图像，（ii）如何在自监督的方式下学习RefSR。特别地，我们提出了一种从双摄像头和多摄像头缩放的观察中进行真实世界RefSR的新型自监督学习方法。首先，考虑到现代智能手机中多个摄像头的流行，更缩放的（望远镜）图像可以自然地作为一个参考，以指导较小缩放（超广角）图像的超分辨率（SR），这给我们机会学习从双缩放观察中进行SR的深度网络。（ii）为了自监督学习DZSR，我们选择望远镜图像作为监督信息，并从它中选择一个中心补丁作为参考，以超分辨率相应的超广角图像补丁。为了减轻在训练过程中超广角低分辨率（LR）补丁与望远镜地面真实（GT）图像之间错位的影响，我们首先采用基于补丁的图像光束对齐，然后设计了一个辅助-LR，以指导失真 LR 特征的变形。为了生成视觉效果好的结果，我们提出了局部重叠的切削韦伯损失，更好地表示 GT 和输出在特征空间中的差异。在测试期间，DZSR可以直接部署用于解决整个超广角图像。此外，我们进一步进行了多次缩放观察，以探索自监督RefSR，并提出了参考图像的有效利用方案。实验结果表明，我们的方法在定量和定性方面都优于现有技术水平。代码可在此处下载：https://www.x剔除。
URL
https://arxiv.org/abs/2405.02171
