Abstract
We generalize the connection between activation function and spline regression/smoothing and characterize how this choice may influence spectral bias within a 1D shallow network. We then demonstrate how gradient descent (GD) can be reinterpreted as a shrinkage operator that masks the singular values of a neural network's Jacobian. Viewed this way, GD implicitly selects the number of frequency components to retain, thereby controlling the spectral bias. An explicit relationship is proposed between the choice of GD hyperparameters (learning rate & number of iterations) and bandwidth (the number of active components). GD regularization is shown to be effective only with monotonic activation functions. Finally, we highlight the utility of non-monotonic activation functions (sinc, Gaussian) as iteration-efficient surrogates for spectral bias.
Abstract (translated)
我们扩展了激活函数与样条回归/平滑之间的联系，并描述了这种选择如何可能影响一维浅层网络中的谱偏差（spectral bias）。然后，我们展示了梯度下降（GD）可以重新解释为一种收缩算子，该算子屏蔽神经网络雅可比矩阵的奇异值。从这个角度来看，GD隐式地选择了要保留的频率分量的数量，从而控制了谱偏差。提出了梯度下降超参数的选择（学习率和迭代次数）与带宽（活动组件数量）之间的显式关系。证明了GD正则化仅在使用单调激活函数时有效。最后，我们强调非单调激活函数（如sinc和高斯）作为具有较高迭代效率的谱偏差替代方案的实用性。
URL
https://arxiv.org/abs/2504.18207
