Abstract
A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients -- gradients through model training -- at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we greatly improve on existing dataset selection methods, outperform accuracy-degrading data poisoning attacks by an order of magnitude, and automatically find competitive learning rate schedules.
Abstract (translated)
训练大规模机器学习模型的一个主要挑战是配置训练过程以最大化模型性能，即从庞大的设计空间中找到最佳的训练设置。在这项工作中，我们解锁了一种基于梯度的方法来解决这个问题。首先，我们引入了一种高效计算元梯度（通过模型训练的梯度）的大规模算法。然后，我们介绍了一个“平滑模型训练”框架，该框架能够利用元梯度进行有效优化。使用元梯度下降（MGD），我们在现有数据集选择方法上取得了显著改进，以一个数量级的优势超越了降低准确性的数据投毒攻击，并且可以自动找到竞争力强的学习率调度。
URL
https://arxiv.org/abs/2503.13751
