Abstract
The stochastic gradient descent (SGD) algorithm has achieved remarkable success in training deep learning models. However, it has several limitations, including susceptibility to vanishing gradients, sensitivity to input data, and a lack of robust theoretical guarantees. In recent years, alternating minimization (AM) methods have emerged as a promising alternative for model training by employing gradient-free approaches to iteratively update model parameters. Despite their potential, these methods often exhibit slow convergence rates. To address this challenge, we propose a novel Triple-Inertial Accelerated Alternating Minimization (TIAM) framework for neural network training. The TIAM approach incorporates a triple-inertial acceleration strategy with a specialized approximation method, facilitating targeted acceleration of different terms in each sub-problem optimization. This integration improves the efficiency of convergence, achieving superior performance with fewer iterations. Additionally, we provide a convergence analysis of the TIAM algorithm, including its global convergence properties and convergence rate. Extensive experiments validate the effectiveness of the TIAM method, showing significant improvements in generalization capability and computational efficiency compared to existing approaches, particularly when applied to the rectified linear unit (ReLU) and its variants.
Abstract (translated)
随机梯度下降（SGD）算法在训练深度学习模型方面取得了显著的成功。然而，它存在几个局限性，包括易受消失梯度的影响、对输入数据的敏感性以及缺乏坚实的理论保证。近年来，交替最小化（AM）方法作为一种有潜力的替代方案出现，通过采用无梯度的方法迭代更新模型参数来训练模型。尽管这些方法具有潜在的优势，但它们通常表现出较慢的收敛速度。为了解决这一挑战，我们提出了一种新颖的三惯性加速交替最小化（Triple-Inertial Accelerated Alternating Minimization, TIAM）框架用于神经网络训练。TIAM 方法采用了一个三惯性加速策略，并结合了专门的近似方法，使不同项在每个子问题优化中的目标加速成为可能。这种集成提高了收敛效率，在较少迭代次数的情况下实现了更好的性能表现。此外，我们还提供了对 TIAM 算法的收敛分析，包括其全局收敛性质和收敛速度。广泛的实验验证了 TIAM 方法的有效性，并表明与现有的方法相比，特别是在应用于修正线性单元（ReLU）及其变体时，TIAM 在泛化能力和计算效率方面取得了显著改进。
URL
https://arxiv.org/abs/2503.08489
