Abstract
Standard gradient descent methods yield point estimates with no measure of confidence. This limitation is acute in overparameterized and low-data regimes, where models have many parameters relative to available data and can easily overfit. Bootstrapping is a classical statistical framework for uncertainty estimation based on resampling, but naively applying it to deep learning is impractical: it requires training many replicas, produces post-hoc estimates that cannot guide learning, and implicitly assumes comparable optima across runs - an assumption that fails in non-convex landscapes. We introduce Twin-Bootstrap Gradient Descent (Twin-Boot), a resampling-based training procedure that integrates uncertainty estimation into optimization. Two identical models are trained in parallel on independent bootstrap samples, and a periodic mean-reset keeps both trajectories in the same basin so that their divergence reflects local (within-basin) uncertainty. During training, we use this estimate to sample weights in an adaptive, data-driven way, providing regularization that favors flatter solutions. In deep neural networks and complex high-dimensional inverse problems, the approach improves calibration and generalization and yields interpretable uncertainty maps.
Abstract (translated)
标准梯度下降方法只能提供点估计，而没有置信度的衡量。这种限制在过度参数化和低数据量的情况下尤为明显，此时模型相对于可用的数据拥有大量参数，并且很容易过拟合。自助法（Bootstrapping）是一种基于重采样的经典统计框架，用于不确定性估计，但直接将其应用于深度学习是不切实际的：它需要训练许多复制模型，产生事后估计无法指导学习过程，而且隐含假设各次运行之间的最优解具有可比性——这一假设在非凸景观中往往不成立。我们引入了双自助梯度下降法（Twin-Bootstrap Gradient Descent, Twin-Boot），这是一种基于重采样的训练程序，它将不确定性估计融入到优化过程中。两个相同的模型并行训练，在独立的自助样本上进行，并通过周期性的均值重置来保持两者在同一个局部最优区域内，从而使它们之间的差异反映该区域内的局部（局域）不确定性。在训练过程中，我们利用这种估计以一种自适应和数据驱动的方式采样权重，提供了一种倾向于更平坦解的正则化方法。这种方法在深度神经网络以及复杂的高维逆问题中改善了校准性和泛化能力，并提供了可解释的不确定性图谱。
URL
https://arxiv.org/abs/2508.15019
