Abstract
The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their plasticity. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies low smoothness. We demonstrate through theoretical analysis and comprehensive experiments that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on the functional properties of transformers. The code is available at this https URL.
Abstract (translated)
Transformer架构的平滑性在泛化、训练稳定性和对抗鲁棒性的背景下已经得到了广泛的研究。然而，它在迁移学习中的作用仍然鲜为人知。在这篇论文中，我们分析了视觉变压器组件适应输入变化的能力，或者换句话说，它们的可塑性。作为平均变化率来定义，它可以捕捉到对输入扰动的敏感度；特别是，高可塑性意味着低平滑度。通过理论分析和全面实验，我们证明这种视角为选择在适应过程中优先考虑的组件提供了原则性的指导。对于从业者而言，一个重要发现是注意力模块和前馈层的高可塑性始终导致更好的微调性能。我们的研究结果与普遍认为平滑性是有益的观点相悖，提出了关于变压器功能属性的新观点。代码可以在[此处](https://this https URL)获取。
URL
https://arxiv.org/abs/2602.06883
