Abstract
We show that the behavior of stochastic gradient descent is related to Bayesian statistics by showing that SGD is effectively diffusion on a fractal landscape, where the fractal dimension can be accounted for in a purely Bayesian way. By doing this we show that SGD can be regarded as a modified Bayesian sampler which accounts for accessibility constraints induced by the fractal structure of the loss landscape. We verify our results experimentally by examining the diffusion of weights during training. These results offer insight into the factors which determine the learning process, and seemingly answer the question of how SGD and purely Bayesian sampling are related.
Abstract (translated)
我们展示了随机梯度下降的行为与贝叶斯统计有关，通过证明SGD在分形景观上的扩散过程，其中分形维度可以通过纯粹的贝叶斯方法来解释。通过这种方式，我们表明SGD可以被视为一种修改后的贝叶斯采样器，该采样器考虑了由损失景观的分形结构引起的可达性限制。我们通过实验验证了这一结果，方法是观察训练过程中权重的扩散情况。这些成果提供了对影响学习过程的因素的理解，并似乎回答了SGD与纯粹的贝叶斯采样之间关系的问题。
URL
https://arxiv.org/abs/2503.22478
