Abstract
Deep Neural Networks (DNNs) are well-known to act as over-parameterized deep image priors (DIP) that regularize various image inverse problems. Meanwhile, researchers also proposed extremely compact, under-parameterized image priors (e.g., deep decoder) that are strikingly competent for image restoration too, despite a loss of accuracy. These two extremes push us to think whether there exists a better solution in the middle: between over- and under-parameterized image priors, can one identify "intermediate" parameterized image priors that achieve better trade-offs between performance, efficiency, and even preserving strong transferability? Drawing inspirations from the lottery ticket hypothesis (LTH), we conjecture and study a novel "lottery image prior" (LIP) by exploiting DNN inherent sparsity, stated as: given an over-parameterized DNN-based image prior, it will contain a sparse subnetwork that can be trained in isolation, to match the original DNN's performance when being applied as a prior to various image inverse problems. Our results validate the superiority of LIPs: we can successfully locate the LIP subnetworks from over-parameterized DIPs at substantial sparsity ranges. Those LIP subnetworks significantly outperform deep decoders under comparably compact model sizes (by often fully preserving the effectiveness of their over-parameterized counterparts), and they also possess high transferability across different images as well as restoration task types. Besides, we also extend LIP to compressive sensing image reconstruction, where a pre-trained GAN generator is used as the prior (in contrast to untrained DIP or deep decoder), and confirm its validity in this setting too. To our best knowledge, this is the first time that LTH is demonstrated to be relevant in the context of inverse problems or image priors.
Abstract (translated)
深度神经网络（DNNs）作为过度参数化的深层图像先验（DIP），已知能够正则化各种图像逆问题。与此同时，研究者也提出了极其紧凑、参数不足的图像先验（例如：深解码器），尽管在准确性上有所损失，但它们对图像恢复同样表现出色。这两种极端促使我们思考是否存在一个更好的中间解决方案：在过度和参数不足的图像先验之间，能否找到实现性能、效率以及保持强迁移能力之间更好平衡的“中间”参数化图像先验？借鉴彩票假设（LTH），我们提出了并研究了一种新颖的“彩票图像先验”（LIP）概念，通过利用DNN内在稀疏性来定义：给定一个基于过度参数化的DNN的图像先验，它将包含一个可以单独训练的稀疏子网络，在应用于各种图像逆问题时可匹配原始DNN的表现。我们的结果验证了LIP的优越性：我们可以在显著的稀疏范围内成功地从过度参数化的DIP中定位到LIP子网络。这些LIP子网络在与深解码器相当紧凑的模型尺寸下表现明显更优（通常完全保留其过度参数化对应物的有效性），并且它们还具备跨不同图像和恢复任务类型的高迁移能力。此外，我们还将LIP扩展到了压缩感知图像重建中，在这里使用预训练的GAN生成器作为先验（与未经过训练的DIP或深解码器相对比），并在这种情况下也证实了其有效性。据我们所知，这是首次在逆问题或图像先验背景下证明LTH的相关性。
URL
https://arxiv.org/abs/2410.24187
