Abstract
Exponential Moving Average (EMA) is a widely used weight averaging (WA) regularization to learn flat optima for better generalizations without extra cost in deep neural network (DNN) optimization. Despite achieving better flatness, existing WA methods might fall into worse final performances or require extra test-time computations. This work unveils the full potential of EMA with a single line of modification, i.e., switching the EMA parameters to the original model after each epoch, dubbed as Switch EMA (SEMA). From both theoretical and empirical aspects, we demonstrate that SEMA can help DNNs to reach generalization optima that better trade-off between flatness and sharpness. To verify the effectiveness of SEMA, we conduct comparison experiments with discriminative, generative, and regression tasks on vision and language datasets, including image classification, self-supervised learning, object detection and segmentation, image generation, video prediction, attribute regression, and language modeling. Comprehensive results with popular optimizers and networks show that SEMA is a free lunch for DNN training by improving performances and boosting convergence speeds.
Abstract (translated)
指数移动平均（EMA）是一种广泛使用的加权平均（WA）正则化方法，用于学习在深度神经网络（DNN）中平滑的优化解，同时不产生额外的优化成本。尽管达到了更好的平滑度，但现有的WA方法可能会陷入更差的最终性能，或者需要额外的测试时间计算。这项工作揭示了EMA的全套潜力，只需对每个epoch进行一次修改，即在每次迭代后将EMA参数更改为原始模型，称之为切换EMA（SEMA）。从理论和实证两个方面来看，我们证明了SEMA可以帮助DNN达到在平滑性和尖度之间进行更好权衡的泛化优化解。为了验证SEMA的有效性，我们在视觉和语言数据集上进行了包括图像分类、自监督学习、目标检测和分割、图像生成、视频预测、属性回归和语言建模等在内的比较实验。使用流行的优化器和网络的结果表明，SEMA是通过提高性能和加快收敛速度来给DNN训练带来免费午餐的好方法。
URL
https://arxiv.org/abs/2402.09240
