Abstract
As learned image compression (LIC) methods become increasingly computationally demanding, enhancing their training efficiency is crucial. This paper takes a step forward in accelerating the training of LIC methods by modeling the neural training dynamics. We first propose a Sensitivity-aware True and Dummy Embedding Training mechanism (STDET) that clusters LIC model parameters into few separate modes where parameters are expressed as affine transformations of reference parameters within the same mode. By further utilizing the stable intra-mode correlations throughout training and parameter sensitivities, we gradually embed non-reference parameters, reducing the number of trainable parameters. Additionally, we incorporate a Sampling-then-Moving Average (SMA) technique, interpolating sampled weights from stochastic gradient descent (SGD) training to obtain the moving average weights, ensuring smooth temporal behavior and minimizing training state variances. Overall, our method significantly reduces training space dimensions and the number of trainable parameters without sacrificing model performance, thus accelerating model convergence. We also provide a theoretical analysis on the Noisy quadratic model, showing that the proposed method achieves a lower training variance than standard SGD. Our approach offers valuable insights for further developing efficient training methods for LICs.
Abstract (translated)
随着学习型图像压缩（LIC）方法计算需求的增加，提高其训练效率变得至关重要。本文通过建模神经网络训练动力学来加速LIC方法的训练过程。我们首先提出了一个基于敏感度感知的真实和虚拟嵌入训练机制（STDET），该机制将LIC模型参数聚类成少数几种不同的模式，在每个模式内，参数可以表示为参考参数的仿射变换。通过进一步利用各模式内部在训练过程中稳定的关联关系及参数敏感性，我们逐步嵌入非参考参数，从而减少可训练参数的数量。 此外，我们还引入了采样后移动平均（SMA）技术，该技术从随机梯度下降(SGD)训练中插值出采样的权重，以此来获取移动平均权重，确保时间行为的平滑性并最小化训练状态方差。总体而言，我们的方法在不牺牲模型性能的前提下显著减少了训练空间维度和可训练参数的数量，从而加速了模型收敛速度。 我们还对噪声二次模型进行了理论分析，表明所提出的方法比标准SGD实现了更低的训练方差。本文的研究为开发更高效的LIC训练方法提供了有价值的见解。
URL
https://arxiv.org/abs/2505.18107
