Abstract
Recently, Transformers have gained significant popularity in image restoration tasks such as image super-resolution and denoising, owing to their superior performance. However, balancing performance and computational burden remains a long-standing problem for transformer-based architectures. Due to the quadratic complexity of self-attention, existing methods often restrict attention to local windows, resulting in limited receptive field and suboptimal performance. To address this issue, we propose Adaptive Token Dictionary (ATD), a novel transformer-based architecture for image restoration that enables global dependency modeling with linear complexity relative to image size. The ATD model incorporates a learnable token dictionary, which summarizes external image priors (i.e., typical image structures) during the training process. To utilize this information, we introduce a token dictionary cross-attention (TDCA) mechanism that enhances the input features via interaction with the learned dictionary. Furthermore, we exploit the category information embedded in the TDCA attention maps to group input features into multiple categories, each representing a cluster of similar features across the image and serving as an attention group. We also integrate the learned category information into the feed-forward network to further improve feature fusion. ATD and its lightweight version ATD-light, achieve state-of-the-art performance on multiple image super-resolution benchmarks. Moreover, we develop ATD-U, a multi-scale variant of ATD, to address other image restoration tasks, including image denoising and JPEG compression artifacts removal. Extensive experiments demonstrate the superiority of out proposed models, both quantitatively and qualitatively.
Abstract (translated)
最近，Transformer模型在图像恢复任务（如超分辨率和去噪）中因其优越的性能而获得了广泛应用。然而，如何平衡性能与计算负担仍是基于Transformer架构的一个长期难题。由于自注意力机制具有二次复杂度，现有方法通常限制注意范围为局部窗口，导致感受野受限且性能不理想。为了应对这一问题，我们提出了适应性令牌字典（Adaptive Token Dictionary, ATD），这是一种新颖的用于图像恢复的Transformer架构，能够在与图像大小呈线性关系的情况下实现全局依赖建模。 ATD模型中引入了一个可学习的令牌字典，在训练过程中它会总结外部图像先验信息（即典型的图像结构）。为了利用这一信息，我们提出了一种令牌字典交叉注意力（Token Dictionary Cross-Attention, TDCA）机制，通过与所学字典互动来增强输入特征。此外，我们还利用了嵌入在TDCA注意图中的类别信息，将输入特征划分为多个类别，每个类别代表图像中相似特征的集群，并作为注意力组使用。同时，我们将学习到的类别信息整合进前馈网络中，以进一步改进特征融合。 ATD及其轻量级版本ATD-light，在多个图像超分辨率基准测试上达到了最先进的性能水平。此外，我们还开发了ATD-U，这是ATD的一个多尺度变体，用于解决包括图像去噪和JPEG压缩痕迹去除在内的其他图像恢复任务。广泛的实验结果从定量和定性两个方面证明了所提出模型的优越性。
URL
https://arxiv.org/abs/2603.02581
