Abstract
Vision Transformer (ViT) models have recently emerged as powerful and versatile models for various visual tasks. Recently, a work called PMF has achieved promising results in few-shot image classification by utilizing pre-trained vision transformer models. However, PMF employs full fine-tuning for learning the downstream tasks, leading to significant overfitting and storage issues, especially in the remote sensing domain. In order to tackle these issues, we turn to the recently proposed parameter-efficient tuning methods, such as VPT, which updates only the newly added prompt parameters while keeping the pre-trained backbone frozen. Inspired by VPT, we propose the Meta Visual Prompt Tuning (MVP) method. Specifically, we integrate the VPT method into the meta-learning framework and tailor it to the remote sensing domain, resulting in an efficient framework for Few-Shot Remote Sensing Scene Classification (FS-RSSC). Furthermore, we introduce a novel data augmentation strategy based on patch embedding recombination to enhance the representation and diversity of scenes for classification purposes. Experiment results on the FS-RSSC benchmark demonstrate the superior performance of the proposed MVP over existing methods in various settings, such as various-way-various-shot, various-way-one-shot, and cross-domain adaptation.
Abstract (translated)
视觉变换器(ViT)模型最近成为各种视觉任务的强大和多功能模型。最近,一个名为PMF的工作在少量图像分类方面取得了令人瞩目的成果,利用预先训练的视觉变换器模型。然而,PMF采用了全 fine-tuning 来学习后续任务,导致严重的过拟合和存储问题,特别是在遥感领域。为了解决这些问题,我们转向了最近提出的参数高效的调整方法,例如 VPT,它只更新新添加的 prompt parameters,而保持预先训练的核心框架冻结。受到 VPT 的启发,我们提出了 Meta Visual Prompt Tuning (MVP) 方法。具体来说,我们将 VPT 方法纳入了元学习框架,并针对遥感领域进行定制,从而生成高效的框架,用于少量遥感场景分类(FS-RSSC)。此外,我们引入了基于补丁嵌入重构的一种新的数据增强策略,以提高场景的表示和多样性,以分类目的为例进行展示。FS-RSSC 基准实验结果证明了所提出的 MVP 在多种设置下比现有方法表现更好,例如不同方式的各种次数、不同方式的一次访问和跨域适应。
URL
https://arxiv.org/abs/2309.09276
