Abstract
We propose InCA, a lightweight method for transfer learning that cross-attends to any activation layer of a pre-trained model. During training, InCA uses a single forward pass to extract multiple activations, which are passed to external cross-attention adapters, trained anew and combined or selected for downstream tasks. We show that, even when selecting a single top-scoring adapter, InCA achieves performance comparable to full fine-tuning, at a cost comparable to fine-tuning just the last layer. For example, with a cross-attention probe 1.3% the size of a pre-trained ViT-L/16 model, we achieve performance within 0.2% of the full fine-tuning paragon at 51% training cost of the baseline, on average across 11 downstream classification tasks. Unlike other forms of efficient adaptation, InCA does not require backpropagating through the pre-trained model, thus leaving its execution unaltered at both training and inference. The versatility of InCA is best illustrated in fine-grained tasks, which may require accessing information absent in the last layer but accessible in intermediate layer activations. Since the backbone is fixed, InCA allows parallel ensembling as well as parallel execution of multiple tasks. InCA achieves state-of-the-art performance in the ImageNet-to-Sketch multi-task benchmark.
Abstract (translated)
我们提出了InCA,一种轻量级的方法,用于迁移学习,它可以跨 attend 训练模型中的任意激活层。在训练中,InCA使用一个 forward 通道提取多个激活,并将其传递给外部跨注意力Adapter,重新训练并组合或为后续任务选择。我们证明,即使选择单个高分Adapter,InCA也能实现与全微调相当的性能,成本与微调最后一个层相当。例如,使用一个跨注意力探测器的大小为训练模型ViT-L/16模型的1.3%,我们在基准线的51%训练成本上,平均情况下对11个后续分类任务达到性能与全微调范式相当。与高效的适应方式其他形式不同,InCA不需要通过训练模型进行反向传播,因此训练和推理时其执行未发生变化。InCA的灵活度最好体现在精细任务上,这些任务可能需要访问最后一个层但没有在中间层可用的信息。由于基线是固定的,InCA允许并行组合和并行执行多个任务。InCA在ImageNet到Sketch多任务基准测试中实现了最先进的性能。
URL
https://arxiv.org/abs/2303.04105
