Abstract
Transfer learning with models pretrained on ImageNet has become a standard practice in computer vision. Transfer learning refers to fine-tuning pretrained weights of a neural network on a downstream task, typically unrelated to ImageNet. However, pretrained weights can become saturated and may yield insignificant gradients, failing to adapt to the downstream task. This hinders the ability of the model to train effectively, and is commonly referred to as loss of neural plasticity. Loss of plasticity may prevent the model from fully adapting to the target domain, especially when the downstream dataset is atypical in nature. While this issue has been widely explored in continual learning, it remains relatively understudied in the context of transfer learning. In this work, we propose the use of a targeted weight re-initialization strategy to restore neural plasticity prior to fine-tuning. Our experiments show that both convolutional neural networks (CNNs) and vision transformers (ViTs) benefit from this approach, yielding higher test accuracy with faster convergence on several image classification benchmarks. Our method introduces negligible computational overhead and is compatible with common transfer learning pipelines.
Abstract (translated)
在ImageNet上预训练的模型进行迁移学习已成为计算机视觉领域的标准做法。迁移学习指的是在后续任务（通常与ImageNet无关）上微调神经网络的预训练权重。然而，预训练权重可能趋于饱和，产生不显著的梯度变化，从而无法适应下游任务。这阻碍了模型的有效训练，通常被称为神经可塑性丧失。神经可塑性丧失可能使模型无法充分适应目标领域，尤其当后续数据集本身具有非典型特征时。尽管该问题在持续学习领域已被广泛研究，但在迁移学习语境中仍相对缺乏探索。本研究提出采用针对性权重重初始化策略，在微调前恢复神经可塑性。实验表明，卷积神经网络（CNNs）和视觉Transformer（ViTs）均能从此方法中受益，在多个图像分类基准数据集上实现更高的测试准确率与更快的收敛速度。该方法几乎不增加计算开销，且与常见迁移学习流程兼容。
URL
https://arxiv.org/abs/2603.20860
