Abstract
LayerNorm is pivotal in Vision Transformers (ViTs), yet its fine-tuning dynamics under data scarcity and domain shifts remain underexplored. This paper shows that shifts in LayerNorm parameters after fine-tuning (LayerNorm shifts) are indicative of the transitions between source and target domains; its efficacy is contingent upon the degree to which the target training samples accurately represent the target domain, as quantified by our proposed Fine-tuning Shift Ratio ($FSR$). Building on this, we propose a simple yet effective rescaling mechanism using a scalar $\lambda$ that is negatively correlated to $FSR$ to align learned LayerNorm shifts with those ideal shifts achieved under fully representative data, combined with a cyclic framework that further enhances the LayerNorm fine-tuning. Extensive experiments across natural and pathological images, in both in-distribution (ID) and out-of-distribution (OOD) settings, and various target training sample regimes validate our framework. Notably, OOD tasks tend to yield lower $FSR$ and higher $\lambda$ in comparison to ID cases, especially with scarce data, indicating under-represented target training samples. Moreover, ViTFs fine-tuned on pathological data behave more like ID settings, favoring conservative LayerNorm updates. Our findings illuminate the underexplored dynamics of LayerNorm in transfer learning and provide practical strategies for LayerNorm fine-tuning.
Abstract (translated)
层归一化（LayerNorm）在视觉变换器（ViT）中起着至关重要的作用，但其在数据稀缺和领域变化情况下的微调动态特性尚未得到充分研究。本文表明，在微调后出现的层归一化参数的变化（即“层归一化偏移”）可以指示源域与目标域之间的转换；它的有效性取决于目标训练样本多大程度上准确代表了目标域，这可以通过我们提出的微调偏移比率($FSR$)来量化。基于此发现，本文提出了一种简单而有效的重缩放机制，该机制使用一个标量$\lambda$（与$FSR$负相关）来对齐学习到的层归一化偏移和完全代表性的数据下理想化的层归一化偏移，并结合了一个循环框架以进一步增强层归一化的微调过程。在自然和病理图像上进行广泛的跨分布（ID）和非分布（OOD）设置以及各种目标训练样本制度下的实验验证了我们的框架的有效性。 值得注意的是，与ID情况相比，特别是在数据稀缺的情况下，OOD任务往往导致较低的$FSR$值和较高的$\lambda$值，这表明了目标训练样本代表性不足。此外，在病理数据上微调后的ViT表现出更接近于ID设置的行为，并倾向于保守地更新层归一化参数。我们的研究揭示了层归一化在迁移学习中被忽视的动力学特性，并为层归一化的微调提供了实用策略。
URL
https://arxiv.org/abs/2508.07577
