Abstract
Numerous lines of aim to control $\textit{model disagreement}$ -- the extent to which two machine learning models disagree in their predictions. We adopt a simple and standard notion of model disagreement in real-valued prediction problems, namely the expected squared difference in predictions between two models trained on independent samples, without any coordination of the training processes. We would like to be able to drive disagreement to zero with some natural parameter(s) of the training procedure using analyses that can be applied to existing training methodologies. We develop a simple general technique for proving bounds on independent model disagreement based on $\textit{anchoring}$ to the average of two models within the analysis. We then apply this technique to prove disagreement bounds for four commonly used machine learning algorithms: (1) stacked aggregation over an arbitrary model class (where disagreement is driven to 0 with the number of models $k$ being stacked) (2) gradient boosting (where disagreement is driven to 0 with the number of iterations $k$) (3) neural network training with architecture search (where disagreement is driven to 0 with the size $n$ of the architecture being optimized over) and (4) regression tree training over all regression trees of fixed depth (where disagreement is driven to 0 with the depth $d$ of the tree architecture). For clarity, we work out our initial bounds in the setting of one-dimensional regression with squared error loss -- but then show that all of our results generalize to multi-dimensional regression with any strongly convex loss.
Abstract (translated)
许多研究工作旨在控制**模型分歧**，即两个机器学习模型在预测中不一致的程度。我们采用了一种关于实值预测问题中的模型分歧的简单且标准的概念：这是指训练过程中，在独立样本上训练的两个模型之间预测结果的期望平方差（无需协调各自的训练过程）。 我们的目标是能够通过调整一些自然参数来将这种分歧降至零，这些参数可以应用于现有的训练方法。为此，我们开发了一种基于**锚定到两个模型平均值**来进行独立模型分歧界限证明的基本技术。然后，我们将此技术应用到四种常用的机器学习算法中以证明分歧界限： 1. **堆叠聚合（stacked aggregation）**：针对任意模型类别的堆叠聚合方法，在该方法中，随着被堆叠的模型数量 $k$ 的增加，分歧趋向于零。 2. **梯度提升（gradient boosting）**：在梯度提升算法中，通过迭代次数 $k$ 的增加来使分歧减少直至趋近于零。 3. **神经网络训练与架构搜索**：对于使用架构搜索进行的神经网络训练，在优化目标为特定大小 $n$ 架构的情况下，随着架构规模的增长，分歧也趋向于减小直到接近于零。 4. **回归树训练（regression tree training）**：当对所有固定深度 $d$ 的回归树进行训练时，随着树的深度增加，模型之间的分歧将减少至趋于零。 为了解释清晰，我们首先在单变量回归和平方误差损失的情况下推导出了初始界限，然后展示了我们的结果可以推广到具有任意强凸损失函数的多维回归问题中。
URL
https://arxiv.org/abs/2602.23360
