Abstract
Image restoration and spectral reconstruction are longstanding computer vision tasks. Currently, CNN-transformer hybrid models provide state-of-the-art performance for these tasks. The key common ingredient in the architectural designs of these models is Channel-wise Self-Attention (CSA). We first show that CSA is an overall low-rank operation. Then, we propose an instance-Guided Low-rank Multi-Head selfattention (GLMHA) to replace the CSA for a considerable computational gain while closely retaining the original model performance. Unique to the proposed GLMHA is its ability to provide computational gain for both short and long input sequences. In particular, the gain is in terms of both Floating Point Operations (FLOPs) and parameter count reduction. This is in contrast to the existing popular computational complexity reduction techniques, e.g., Linformer, Performer, and Reformer, for whom FLOPs overpower the efficient design tricks for the shorter input sequences. Moreover, parameter reduction remains unaccounted for in the existing this http URL perform an extensive evaluation for the tasks of spectral reconstruction from RGB images, spectral reconstruction from snapshot compressive imaging, motion deblurring, and image deraining by enhancing the best-performing models with our GLMHA. Our results show up to a 7.7 Giga FLOPs reduction with 370K fewer parameters required to closely retain the original performance of the best-performing models that employ CSA.
Abstract (translated)
图像修复和光谱重建是计算机视觉中的长期任务。目前，CNN-Transformer混合模型为这些任务提供了最先进的性能。这些模型的关键共同组件是通道级自注意力（CSA）。我们首先证明，CSA是一个总的低秩操作。然后，我们提出了一种实例引导的低秩多头自注意力（GLMHA）来代替CSA，从而实现显著的计算开销，同时保留原始模型的性能。与提出的GLMHA独特之处在于其能够为短和长输入序列提供计算开销。特别地，增益既涉及浮点操作（FLOPs），也涉及参数计数减少。这与现有流行的计算复杂度减少技术（例如LINformer、Performer和Reformer）不同，后者的FLOPs超过了对于较短输入序列的高效设计技巧。此外，现有的方法中尚未考虑参数减少。通过增强我们的GLMHA，我们对来自RGB图像的光谱重建、来自快照压缩成像的光谱重建、运动去雾和图像去雨等任务进行了广泛的评估。我们的结果表明，通过使用GLMHA，可以实现最高7.7 Giga FLOPs的减少，同时只需370K个参数，从而保留最佳模型的原始性能。
URL
https://arxiv.org/abs/2410.00380
