Abstract
Memory-efficient training of deep neural networks has become increasingly important as models grow larger while deployment environments impose strict resource constraints. We propose TraDy, a novel transfer learning scheme leveraging two key insights: layer importance for updates is architecture-dependent and determinable a priori, while dynamic stochastic channel selection provides superior gradient approximation compared to static approaches. We introduce a dynamic channel selection approach that stochastically resamples channels between epochs within preselected layers. Extensive experiments demonstrate TraDy achieves state-of-the-art performance across various downstream tasks and architectures while maintaining strict memory constraints, achieving up to 99% activation sparsity, 95% weight derivative sparsity, and 97% reduction in FLOPs for weight derivative computation.
Abstract (translated)
随着模型规模的增大，而在部署环境中资源受限的情况下，深度神经网络的有效训练变得越来越重要。我们提出了一种名为TraDy的新颖迁移学习方案，该方案基于两个关键见解：层的重要性对于更新而言是架构依赖且可以预先确定的；动态随机通道选择提供比静态方法更优越的梯度近似效果。 TraDy引入了一个动态通道选择的方法，在选定的层之间通过在不同epoch间随机重采样通道来实现。这一方案使得模型能够在保持严格内存限制的同时，跨各种下游任务和架构达到最先进的性能，并实现了高达99%的激活稀疏性、95%的权重导数稀疏性和用于权重导数计算的FLOPs减少了97%，从而大幅降低了内存使用量和计算需求。
URL
https://arxiv.org/abs/2510.19675
