Abstract
Theoretical works on supervised transfer learning (STL) -- where the learner has access to labeled samples from both source and target distributions -- have for the most part focused on statistical aspects of the problem, while efficient optimization has received less attention. We consider the problem of designing an SGD procedure for STL that alternates sampling between source and target data, while maintaining statistical transfer guarantees without prior knowledge of the quality of the source data. A main algorithmic difficulty is in understanding how to design such an adaptive sub-sampling mechanism at each SGD step, to automatically gain from the source when it is informative, or bias towards the target and avoid negative transfer when the source is less informative. We show that, such a mixed-sample SGD procedure is feasible for general prediction tasks with convex losses, rooted in tracking an abstract sequence of constrained convex programs that serve to maintain the desired transfer guarantees. We instantiate these results in the concrete setting of linear regression with square loss, and show that the procedure converges, with $1/\sqrt{T}$ rate, to a solution whose statistical performance on the target is adaptive to the a priori unknown quality of the source. Experiments with synthetic and real datasets support the theory.
Abstract (translated)
关于监督迁移学习（STL）的理论工作——在这种情况下，学习者可以从源分布和目标分布中获取标记样本——大多数研究主要集中在问题的统计方面，而高效的优化方法则受到较少的关注。我们考虑设计一种针对STL的SGD过程，该过程在源数据和目标数据之间交替采样，并且在没有先验知识的情况下维持统计迁移保证。算法的一个重要困难在于如何设计适应性的子抽样机制，在每一步的SGD过程中，当源数据具有信息量时可以自动从中获益；而当源数据不太具信息量时，则偏向于目标并避免负面转移。 我们展示了一种混合样本SGD过程对于一般预测任务（使用凸损失函数）是可行的，并且该过程根植于跟踪一系列抽象的约束凸程序，这些程序有助于保持所需的迁移保证。我们在具体的线性回归和平方损失环境中实现了这些结果，并证明了这种程序以$1/\sqrt{T}$的速度收敛至一个统计性能适应未知源质量的目标解。 通过合成数据集和真实数据集进行的实验支持了这一理论。
URL
https://arxiv.org/abs/2507.04194
