Abstract
Self-supervised pre-training using unlabeled data is widely used in machine learning. In this paper, we propose a new self-supervised pre-training approach to dealing with heterogeneous data. Instead of mixing all the data and minimizing the averaged global loss in the conventional way, we impose additional equilibrium constraints to ensure that the models optimizes each source of heterogeneous data to its local optima after $K$-step gradient descent initialized from the model. We formulate this as a bilevel optimization problem, and use the first-order approximation method to solve the problem. We discuss its connection to model-agnostic meta learning (MAML). Experiments are carried out on self-supervised pre-training using multi-domain and multilingual datasets, demonstrating that the proposed approach can significantly improve the adaptivity of the self-supervised pre-trained model for the downstream supervised fine-tuning tasks.
Abstract (translated)
无监督预训练在使用未标记数据的机器学习中广泛应用。本文提出了一种新的自监督预训练方法，用于处理异构数据。不同于传统方法将所有数据混合并最小化平均全局损失，我们引入了额外的平衡约束条件，以确保模型从初始化开始经过 $K$ 步梯度下降后能够优化每一种异构数据到其局部最优解。我们将此问题形式化为双层优化问题，并使用一阶近似方法来求解。讨论了该方法与无模型元学习（MAML）的联系。在多领域和多语言数据集上的自监督预训练实验表明，所提出的这种方法可以显著提高自监督预训练模型对下游有监督微调任务的适应性。
URL
https://arxiv.org/abs/2508.19990
