Abstract
Modern GPUs are equipped with large amounts of high-bandwidth memory, enabling them to support mini-batch sizes of up to tens of thousands of training samples. However, most existing optimizers struggle to perform effectively at such a large batch size. As batch size increases, gradient noise decreases due to averaging over many samples, limiting the ability of first-order methods to escape sharp or suboptimal minima and reach the global minimum. Meanwhile, second-order methods like the natural gradient with Kronecker-Factored Approximate Curvature (KFAC) often require excessively high damping to remain stable at large batch sizes. This high damping effectively washes out the curvature information that gives these methods their advantage, reducing their performance to that of simple gradient descent. In this paper, we introduce Fisher-Orthogonal Projection (FOP), a novel technique that restores the effectiveness of the second-order method at very large batch sizes, enabling scalable training with improved generalization and faster convergence. FOP constructs a variance-aware update direction by leveraging gradients from two sub-batches, enhancing the average gradient with a component of the gradient difference that is orthogonal to the average under the Fisher-metric.
Abstract (translated)
现代GPU配备了大量高带宽内存，能够支持多达数万个训练样本的小批量处理。然而，大多数现有的优化器在如此大的批量大小下难以有效运行。随着批处理量的增加，由于大量样本平均效应，梯度噪声减少，这限制了一阶方法（如随机梯度下降）从尖锐或次优极小值中逃脱并达到全局最小值的能力。同时，像自然梯度结合Kronecker-Factored Approximate Curvature (KFAC)这样的二阶方法，在处理大规模批量时往往需要过高的阻尼以保持稳定。这种高阻尼会抹去这些方法所依赖的曲率信息优势，使它们的表现与简单的梯度下降相差无几。 在本文中，我们介绍了Fisher-Orthogonal Projection（FOP），这是一种新颖的技术，能够在非常大的批处理大小下恢复二阶方法的有效性，从而实现具有改进泛化能力和更快收敛速度的大规模训练。FOP通过利用两个子批量的梯度来构造一个方差感知更新方向，并在Fisher度量下的平均梯度中增加了一个正交于平均值的梯度差异分量。
URL
https://arxiv.org/abs/2508.13898
