Abstract
Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation this http URL, there lacks a unified, rigorous evaluation protocol that is both challenging and realistic for real-world usage. In this work, we establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets, and propose the Hyperparameter Ensemble (HPE) protocol to overcome the "validation set illusion" in data-scarce regimes. Our empirical findings demonstrate that the choice of pre-trained model is the dominant factor for performance, while many sophisticated transfer methods offer negligible practical advantages over a simple full-parameter fine-tuning baseline. To explain this surprising effectiveness, we provide an in-depth mechanistic analysis showing that full fine-tuning succeeds via distributed micro-adjustments and more flexible reshaping of high-level semantic presentations without suffering from overfitting. Additionally, we quantify the performance collapse of multimodal models in specialized domains as a result of linguistic rarity using adjusted Zipf frequency scores. By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research. We make the FEWTRANS benchmark publicly available at this https URL.
Abstract (translated)
少量样本迁移学习（Few-shot Transfer Learning）已通过更强的预训练模型和改进的适应方法取得了显著进展，但在实际应用场景中，仍然缺乏一种统一且严格的评估协议。本文提出了FEWTRANS，这是一个包含10个多样化数据集的全面基准，并引入了超参数集成（Hyperparameter Ensemble, HPE）协议来克服在样本稀缺场景下的“验证集幻觉”问题。我们的实证研究表明，在性能方面预训练模型的选择是决定性因素，而许多复杂的迁移学习方法相比简单的全参数微调基线并未提供实际的优势。为了说明这种意外的有效性，我们进行了深入的机制分析，展示了全参数微调通过分布式细粒度调整和更灵活地重塑高级语义表示来取得成功，并且不会过度拟合。此外，我们使用调整后的Zipf频率分数量化了多模态模型在特定领域中的性能崩溃，这是由于语言稀有性所导致的。 为了为少量样本迁移学习研究提供一个严格的“标尺”，促进可重复的研究进展，我们将FEWTRANS基准公开发布在这个网址：[这里应该是原链接]。通过这一举措，我们希望能够推动该领域的进一步发展和标准化。
URL
https://arxiv.org/abs/2603.00478
