Abstract
Deep learning-based image restoration methods have achieved promising performance. However, how to faithfully preserve the structure of the original image remains challenging. To address this challenge, we propose a novel Residual-Conditioned Optimal Transport (RCOT) approach, which models the image restoration as an optimal transport (OT) problem for both unpaired and paired settings, integrating the transport residual as a unique degradation-specific cue for both the transport cost and the transport map. Specifically, we first formalize a Fourier residual-guided OT objective by incorporating the degradation-specific information of the residual into the transport cost. Based on the dual form of the OT formulation, we design the transport map as a two-pass RCOT map that comprises a base model and a refinement process, in which the transport residual is computed by the base model in the first pass and then encoded as a degradation-specific embedding to condition the second-pass restoration. By duality, the RCOT problem is transformed into a minimax optimization problem, which can be solved by adversarially training neural networks. Extensive experiments on multiple restoration tasks show the effectiveness of our approach in terms of both distortion measures and perceptual quality. Particularly, RCOT restores images with more faithful structural details compared to state-of-the-art methods.
Abstract (translated)
基于深度学习的图像修复方法已经取得了很好的性能。然而，如何忠实保留原始图像的结构仍然具有挑战性。为解决这个问题，我们提出了一个新颖的残差约束优化传输（RCOT）方法，将图像修复建模为对于未配对和成对设置的优化传输（OT）问题，将传输残差作为传输成本和传输映射的唯一退化特定提示。具体来说，我们首先通过将残差的退化特定信息融入传输成本中，形式化了一个Fourier残差引导的OT目标。基于OT公式的双形式，我们设计了一个包含基模型和优化过程的两层RCOT映射，其中传输残差在第一层由基模型计算，然后用退化特定编码作为第二层修复的调节。通过极值，RCOT问题转化为一个最小最大优化问题，可以被对抗性训练的神经网络求解。在多个修复任务上进行的大量实验证明了我们方法在失真度和感知质量方面的有效性。特别是，RCOT修复的图像具有比现有方法更忠实于结构的细节。
URL
https://arxiv.org/abs/2405.02843
