Abstract
Large language models are increasingly deployed across diverse applications. This often includes tasks LLMs have not encountered during training. This implies that enumerating and obtaining the high-quality training data for all tasks is infeasible. Thus, we often need to rely on transfer learning using datasets with different characteristics, and anticipate out-of-distribution requests. Motivated by this practical need, we propose an analysis framework, building a transfer learning matrix and dimensionality reduction, to dissect these cross-task interactions. We train and analyze 10 models to identify latent abilities (e.g., Reasoning, Sentiment Classification, NLU, Arithmetic) and discover the side effects of the transfer learning. Our findings reveal that performance improvements often defy explanations based on surface-level dataset similarity or source data quality. Instead, hidden statistical factors of the source dataset, such as class distribution and generation length proclivities, alongside specific linguistic features, are actually more influential. This work offers insights into the complex dynamics of transfer learning, paving the way for more predictable and effective LLM adaptation.
Abstract (translated)
大型语言模型在各种应用中越来越广泛地部署。这通常包括那些在训练过程中未遇到的任务。这意味着为所有任务枚举并获取高质量的训练数据是不可行的。因此，我们常常需要依赖于使用具有不同特征的数据集进行迁移学习，并且要预料到分布外请求的情况。出于这一实际需求，我们提出了一种分析框架，构建一个转移学习矩阵和维度缩减，以剖析这些跨任务交互。通过训练并分析10个模型来识别潜在的能力（如推理、情感分类、自然语言理解、算术），我们发现迁移学习的副作用。我们的研究结果表明，性能提升往往无法用基于表面数据集相似性或源数据质量的解释来充分说明。相反，源数据集中隐藏的统计因素，例如类别分布和生成长度倾向，以及特定的语言特征实际上更具影响力。这项工作提供了对迁移学习复杂动态的理解，并为更可预测且有效的大型语言模型适应铺平了道路。
URL
https://arxiv.org/abs/2509.13624
