Abstract
Gradient descent typically converges to a single minimum of the training loss without mechanisms to explore alternative minima that may generalize better. Searching for diverse minima directly in high-dimensional parameter space is generally intractable. To address this, we propose a framework that performs training in two distinct phases: search in a tractable representation space (the space of intermediate activations) to find diverse representational solutions, and gradient-based learning in parameter space by regressing to those searched representations. Through evolutionary search, we discover representational solutions whose fitness and diversity scale with compute--larger populations and more generations produce better and more varied solutions. These representations prove to be learnable: networks trained by regressing to searched representations approach SGD's performance on MNIST, CIFAR-10, and CIFAR-100. Performance improves with search compute up to saturation. The resulting models differ qualitatively from networks trained with gradient descent, following different representational trajectories during training. This work demonstrates how future training algorithms could overcome gradient descent's exploratory limitations by decoupling search in representation space from efficient gradient-based learning in parameter space.
Abstract (translated)
梯度下降通常会收敛到训练损失中的单一最小值，而没有探索可能更好泛化的替代极小值的机制。在高维参数空间中直接搜索多样性的极小值通常是不可行的。为了解决这个问题，我们提出了一种框架，在两个不同的阶段进行训练：首先在可处理的表现形式空间（中间激活的空间）中寻找多样化的表示解；然后通过基于梯度的学习方法在参数空间中回归到这些找到的表示上进行学习。 通过进化搜索，我们可以发现那些其适应性和多样性与计算资源成比例增长的表示解——更大的种群和更多的世代会产生更好且更加多样的解决方案。这些表示证明是可学得的：通过回归至搜索到的表示来训练的网络在MNIST、CIFAR-10和CIFAR-100数据集上的性能与随机梯度下降法（SGD）相当。随着搜索计算资源的增加，性能会提升直至饱和。 由此产生的模型在训练过程中遵循不同的表示轨迹，这区别于通过梯度下降方法训练得到的网络。这项工作展示了未来训练算法如何能够通过将表示空间中的搜索和参数空间中基于梯度的学习解耦来克服梯度下降探索上的限制。
URL
https://arxiv.org/abs/2509.10973
