Abstract
Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks. However, full finetuning such a video model could be computationally expensive and unnecessary, given the pre-trained image transformer models have demonstrated exceptional transferability. In this work, we propose a novel method to Adapt pre-trained Image Models (AIM) for efficient video understanding. By freezing the pre-trained image model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal adaptation and joint adaptation to gradually equip an image model with spatiotemporal reasoning capability. We show that our proposed AIM can achieve competitive or even better performance than prior arts with substantially fewer tunable parameters on four video action recognition benchmarks. Thanks to its simplicity, our method is also generally applicable to different image pre-trained models, which has the potential to leverage more powerful image foundation models in the future. The project webpage is \url{this https URL}.
Abstract (translated)
最近的视觉Transformer基于视频模型大多遵循“图像预训练然后微调”范式,并在多个视频基准上取得了巨大成功。然而,完全微调这样的视频模型可能会计算代价昂贵且没有必要,因为预训练的图像Transformer模型已经证明了出色的可转移性。在这项工作中,我们提出了一种新的方法来适应预训练的图像模型(AIM),以高效视频理解。通过冻结预训练图像模型并添加几个轻量级适配器,我们引入了空间适应、时间适应和联合适应,逐渐为图像模型配备时间和空间推理能力。我们表明,我们提出的AIM可以在四个视频动作识别基准上实现竞争或甚至更好的性能,而可调整参数数量大大减少。由于它的简单性,我们的方法也适用于不同的预训练图像模型,这有可能在未来的工作中利用更强大的图像基元模型。项目页面如下所示: url{this https URL}。
URL
https://arxiv.org/abs/2302.03024
