Abstract
Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in small models such expert solutions occupy a negligible fraction of the volume of this distribution, making their discovery reliant on structured optimization methods such as gradient descent. In contrast, in large, well-pretrained models the density of task-experts increases dramatically, so that diverse, task-improving specialists populate a substantial fraction of the neighborhood around the pretrained weights. Motivated by this perspective, we explore a simple, fully parallel post-training method that samples $N$ parameter perturbations at random, selects the top $K$, and ensembles predictions via majority vote. Despite its simplicity, this approach is competitive with standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.
Abstract (translated)
预训练产生的参数向量通常被视作进一步迭代适应的起点。然而，在这项工作中，我们将预训练的结果视为参数向量上的分布，该分布的支持集中已包含了特定任务的专家模型。我们展示了在小规模模型中，这些专家解决方案所占的空间非常微小，因此其发现依赖于结构化的优化方法如梯度下降法。相比之下，在大规模且充分预训练的模型中，任务相关专家的数量显著增加，多样而有效的专家占据了预训练权重附近很大一部分区域。 基于这一视角，我们探索了一种简单的、完全并行的后训练方法：随机抽取$N$个参数扰动，选择其中前$K$个，并通过多数投票的方式进行集成预测。尽管该方法简单，它在当代大规模模型中的表现仍可与标准的后训练方法（如PPO, GRPO和ES）相媲美。
URL
https://arxiv.org/abs/2603.12228
