Abstract
We consider the problem of generalization of arbitrarily overparameterized two-layer ReLU Neural Networks with univariate input. Recent work showed that under square loss, flat solutions (motivated by flat / stable minima and Edge of Stability phenomenon) provably cannot overfit, but it remains unclear whether the same phenomenon holds for logistic loss. This is a puzzling open problem because existing work on logistic loss shows that gradient descent with increasing step size converges to interpolating solutions (at infinity, for the margin-separable cases). In this paper, we prove that the \emph{flatness implied generalization} is more delicate under logistic loss. On the positive side, we show that flat solutions enjoy near-optimal generalization bounds within a region between the left-most and right-most \emph{uncertain} sets determined by each candidate solution. On the negative side, we show that there exist arbitrarily flat yet overfitting solutions at infinity that are (falsely) certain everywhere, thus certifying that flatness alone is insufficient for generalization in general. We demonstrate the effects predicted by our theory in a well-controlled simulation study.
Abstract (translated)
我们考虑了单变量输入的任意过参数化两层ReLU神经网络的泛化问题。最近的研究表明，在平方损失的情况下，平坦解（受稳定极小值和平稳性边界现象启发）可以被证明不会导致过拟合，但对于对数几率损失而言，这种情况是否仍然成立尚不清楚。这是一个令人困惑的未解决问题，因为现有的关于对数几率损失的工作显示，随着步长增加，梯度下降会收敛到插值解（在边际可分的情况下，在无穷远处）。在这篇论文中，我们证明了在对数几率损失下，“平坦性引起的泛化”更加微妙。从积极的一面来看，我们在由每个候选解确定的最左和最右“不确定集”之间的区域内展示了平坦解几乎具有最优的泛化界。另一方面，我们也表明存在一些解决方案，它们在无穷远处是任意平坦的且会导致过拟合，并且它们在整个范围内被认为是（错误地）确定性的，从而证明了仅仅依靠平坦性不足以保证泛化能力的一般情形。我们在一项严格控制的模拟研究中展示了我们理论预测的影响。
URL
https://arxiv.org/abs/2512.01473
