Abstract
Transfer learning is a cornerstone of modern machine learning, promising a way to adapt models pretrained on a broad mix of data to new tasks with minimal new data. However, a significant challenge remains in ensuring that transferred features are sufficient to handle unseen datasets, amplified by the difficulty of quantifying whether two tasks are "related". To address these challenges, we evaluate model transfer from a pretraining mixture to each of its component tasks, assessing whether pretrained features can match the performance of task-specific direct training. We identify a fundamental limitation in deep learning models -- an "information saturation bottleneck" -- where networks fail to learn new features once they encode similar competing features during training. When restricted to learning only a subset of key features during pretraining, models will permanently lose critical features for transfer and perform inconsistently on data distributions, even components of the training mixture. Empirical evidence from published studies suggests that this phenomenon is pervasive in deep learning architectures -- factors such as data distribution or ordering affect the features that current representation learning methods can learn over time. This study suggests that relying solely on large-scale networks may not be as effective as focusing on task-specific training, when available. We propose richer feature representations as a potential solution to better generalize across new datasets and, specifically, present existing methods alongside a novel approach, the initial steps towards addressing this challenge.
Abstract (translated)
迁移学习是现代机器学习的基石，它承诺了一种方法，可以将预先训练好的模型在大量数据上进行微调，以便适应新的任务，并且只需要少量的新数据。然而，一个重要的挑战仍然存在：如何确保迁移到新任务中的特征足以应对未见过的数据集，尤其是在量化两个任务是否“相关”这一难题面前。 为了解决这些问题，我们评估了从预训练混合物到其各个组成部分任务的模型迁移情况，以确定预先训练好的特征能否达到针对特定任务直接训练的表现水平。我们的研究揭示了一个深度学习模型的基本局限性——一种“信息饱和瓶颈”，在这种情况下，网络一旦在训练过程中编码了相似的竞争特征就无法再学习新的特征。 当我们限制模型仅在预训练阶段学习关键特征的一个子集时，发现这些模型会永久失去对于迁移至关重要的某些特征，并且会在数据分布上表现不稳定，甚至包括它们原来的数据混合部分。来自已发表研究的实证证据表明，这种现象广泛存在于深度学习架构中——数据分布或顺序等因素影响了当前表示学习方法随时间所能学到的特征。 这项研究表明，在有特定任务训练的情况下，仅仅依赖大规模网络可能不如专注于针对具体任务的训练有效。我们提出更丰富的特征表示作为解决这一挑战的一种潜在方案，以更好地在新数据集中泛化，并且特别地介绍了现有方法以及一种新的方法，这标志着应对这一挑战的第一步。
URL
https://arxiv.org/abs/2506.18221
