Abstract
Sharpness-Aware Minimization (SAM) improves model generalization but doubles the computational cost of Stochastic Gradient Descent (SGD) by requiring twice the gradient calculations per optimization step. To mitigate this, we propose Adaptively sampling-Reusing-mixing decomposed gradients to significantly accelerate SAM (ARSAM). Concretely, we firstly discover that SAM's gradient can be decomposed into the SGD gradient and the Projection of the Second-order gradient onto the First-order gradient (PSF). Furthermore, we observe that the SGD gradient and PSF dynamically evolve during training, emphasizing the growing role of the PSF to achieve a flat minima. Therefore, ARSAM is proposed to the reused PSF and the timely updated PSF still maintain the model's generalization ability. Extensive experiments show that ARSAM achieves state-of-the-art accuracies comparable to SAM across diverse network architectures. On CIFAR-10/100, ARSAM is comparable to SAM while providing a speedup of about 40\%. Moreover, ARSAM accelerates optimization for the various challenge tasks (\textit{e.g.}, human pose estimation, and model quantization) without sacrificing performance, demonstrating its broad practicality.% The code is publicly accessible at: this https URL.
Abstract (translated)
Sharpness-Aware Minimization (SAM) enhances model generalization but doubles the computational cost of Stochastic Gradient Descent (SGD) by necessitating twice as many gradient calculations per optimization step. To address this issue, we propose Adaptively Sampling-Reusing-Mixing Decomposed Gradients to significantly accelerate SAM (ARSAM). Specifically, we first discover that SAM's gradients can be decomposed into the SGD gradient and the Projection of the Second-order Gradient onto the First-order Gradient (PSF). Additionally, we observe that both the SGD gradient and PSF dynamically change during training, with the role of PSF becoming increasingly significant to achieve a flatter minimum. ARSAM is designed to effectively reuse old PSFs while ensuring that newly updated PSFs maintain the model's generalization capabilities. Extensive experiments demonstrate that ARSAM achieves state-of-the-art accuracy comparable to SAM across various network architectures. On datasets such as CIFAR-10 and CIFAR-100, ARSAM matches SAM in performance but offers a 40% speedup. Furthermore, ARSAM accelerates optimization for diverse challenging tasks (e.g., human pose estimation and model quantization) without compromising performance, showcasing its broad practical applicability. (Note: The original text mentions that the code is publicly accessible at a specific URL. If you need the link to be included or have another way of accessing it, please provide additional details.)
URL
https://arxiv.org/abs/2510.03763
